CategoryUncategorized

OVN Routing and ovn-trace

In the last post we covered how OVN distributes the pipeline execution across the different hypervisors thanks to the encapsulation used. Using the same system as a base, let's create a Logical Router and a second Logical Switch where we can see how routing works between different nodes.

 

ovn-nbctl ls-add network1
ovn-nbctl ls-add network2
ovn-nbctl lsp-add network1 vm1
ovn-nbctl lsp-add network2 vm2
ovn-nbctl lsp-set-addresses vm1 "40:44:00:00:00:01 192.168.0.11"
ovn-nbctl lsp-set-addresses vm2 "40:44:00:00:00:02 192.168.1.11"

ovn-nbctl lr-add router1
ovn-nbctl lrp-add router1 router1-net1 40:44:00:00:00:03 192.168.0.1/24
ovn-nbctl lsp-add network1 net1-router1
ovn-nbctl lsp-set-addresses net1-router1 40:44:00:00:00:03
ovn-nbctl lsp-set-type net1-router1 router
ovn-nbctl lsp-set-options net1-router1 router-port=router1-net1

ovn-nbctl lrp-add router1 router1-net2 40:44:00:00:00:04 192.168.1.1/24
ovn-nbctl lsp-add network2 net2-router1
ovn-nbctl lsp-set-addresses net2-router1 40:44:00:00:00:04
ovn-nbctl lsp-set-type net2-router1 router
ovn-nbctl lsp-set-options net2-router1 router-port=router1-net2

And then on Worker1 and Worker2 respectively, let's bind VM1 and VM2 ports inside network namespaces:

# Worker1
ovs-vsctl add-port br-int vm1 -- set Interface vm1 type=internal -- set Interface vm1 external_ids:iface-id=vm1
ip netns add vm1
ip link set vm1 netns vm1
ip netns exec vm1 ip link set vm1 address 40:44:00:00:00:01
ip netns exec vm1 ip addr add 192.168.0.11/24 dev vm1
ip netns exec vm1 ip link set vm1 up
ip netns exec vm1 ip route add default via 192.168.0.1

# Worker2
ovs-vsctl add-port br-int vm2 -- set Interface vm2 type=internal -- set Interface vm2 external_ids:iface-id=vm2
ip netns add vm2
ip link set vm2 netns vm2
ip netns exec vm2 ip link set vm2 address 40:44:00:00:00:02
ip netns exec vm2 ip addr add 192.168.1.11/24 dev vm2
ip netns exec vm2 ip link set vm2 up
ip netns exec vm2 ip route add default via 192.168.1.1

First, let's check connectivity between two VMs through the Logical Router:

[root@worker1 ~]# ip netns exec vm1 ping 192.168.1.11 -c2
PING 192.168.1.11 (192.168.1.11) 56(84) bytes of data.
64 bytes from 192.168.1.11: icmp_seq=1 ttl=63 time=0.371 ms
64 bytes from 192.168.1.11: icmp_seq=2 ttl=63 time=0.398 ms

--- 192.168.1.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.371/0.384/0.398/0.023 ms

ovn-trace

Let's now check with ovn-trace how the flow of a packet from VM1 to VM2 looks like and then we'll verify how this gets realized in our physical deployment with 3 hypervisors: 

[root@central ~]# ovn-trace --summary network1 'inport == "vm1" && eth.src == 40:44:00:00:00:01 && eth.dst == 40:44:00:00:00:03 && ip4.src == 192.168.0.11 && ip4.dst == 192.168.1.11 && ip.ttl == 64'
# ip,reg14=0x1,vlan_tci=0x0000,dl_src=40:44:00:00:00:01,dl_dst=40:44:00:00:00:03,nw_src=192.168.0.11,nw_dst=192.168.1.11,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=64
ingress(dp="network1", inport="vm1") {
    next;
    outport = "net1-router1";
    output;
    egress(dp="network1", inport="vm1", outport="net1-router1") {
        output;
        /* output to "net1-router1", type "patch" */;
        ingress(dp="router1", inport="router1-net1") {
            next;
            ip.ttl--;
            reg0 = ip4.dst;
            reg1 = 192.168.1.1;
            eth.src = 40:44:00:00:00:04;
            outport = "router1-net2";
            flags.loopback = 1;
            next;
            eth.dst = 40:44:00:00:00:02;
            next;
            output;
            egress(dp="router1", inport="router1-net1", outport="router1-net2") {
                output;
                /* output to "router1-net2", type "patch" */;
                ingress(dp="network2", inport="net2-router1") {
                    next;
                    outport = "vm2";
                    output;
                    egress(dp="network2", inport="net2-router1", outport="vm2") {
                        output;
                        /* output to "vm2", type "" */;
                    };
                };
            };
        };
    };
};

As you can see, the packet goes through 3 OVN datapaths: network1, router1 and network2. As VM1 is on Worker1 and VM2 is on Worker2, the packet will traverse the tunnel between both hypervisors and thanks to the Geneve encapsulation, the execution pipeline will be distributed:

  • Worker1 will execute both ingress and egress pipelines of network1
  • Worker1 will also perform the routing (lines 10 to 28 above) and the ingress pipeline for network2. Then the packet will be pushed to Worker2 via the tunnel.
  • Worker2 will execute the egress pipeline for network2 delivering the packet to VM2.

Let's launch a ping from VM1 and check the Geneve traffic on Worker2: 

52:54:00:13:e0:a2 > 52:54:00:ac:67:5b, ethertype IPv4 (0x0800), length 156: (tos 0x0, ttl 64, id 46587, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.50.100.55145 > 192.168.50.101.6081: [bad udp cksum 0xe6a5 -> 0xb087!] Geneve, Flags [C],
vni 0x8,
proto TEB (0x6558),
options [class Open Virtual Networking (OVN) (0x102)
type 0x80(C) len 8
data 00020001]

From what we learnt in the previous post, when the packet arrives to Worker2, the egress pipeline of Datapath 8 (VNI) will be executed being ingress port = 2 and egress port = 1. Let's see if it matches what ovn-trace gave us earlier:

egress(dp="network2", inport="net2-router1", outport="vm2") 

[root@central ~]# ovn-sbctl get Datapath_Binding network2 tunnel_key
8
[root@central ~]# ovn-sbctl get Port_Binding net2-router1 tunnel-key
2
[root@central ~]# ovn-sbctl get Port_Binding vm2 tunnel-key
1

What about the reply from VM2 to VM1? If we check the tunnel traffic on Worker2 we'll see the ICMP echo reply packets with different datapath and ingress/egress ports:

22:19:03.896340 52:54:00:ac:67:5b > 52:54:00:13:e0:a2, ethertype IPv4 (0x0800), length 156: (tos 0x0, ttl 64, id 30538, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.50.101.13530 > 192.168.50.100.6081: [bad udp cksum 0xe6a5 -> 0x5419!] Geneve, Flags [C],
vni 0x7,
proto TEB (0x6558),
options [class Open Virtual Networking (OVN) (0x102)
type 0x80(C) len 8
data 00020001]


[root@central ~]# ovn-sbctl get Datapath_Binding network1 tunnel_key
7
[root@central ~]# ovn-sbctl get Port_Binding net1-router1 tunnel-key
2
[root@central ~]# ovn-sbctl get Port_Binding vm1 tunnel-key
1

As we can see, the routing happens locally in the source node; there's no need to send the packet to any central/network node as the East/West routing is always distributed with OVN. In the case of North/South traffic, where SNAT is required, traffic will go to the gateway node but we'll cover this in coming posts.

OpenFlow analysis

Until now, we just focused on the Logical elements but ovn-controller will ultimately translate the Logical Flows into physical flows on the OVN bridge.  Let's inspect the ports that our bridge has on Worker2:

[root@worker2 ~]# ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000404400000002
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(ovn-centra-0): addr:ea:64:c2:a9:86:fe
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(ovn-worker-0): addr:de:96:64:2b:21:4a
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 8(vm2): addr:40:44:00:00:00:02
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-int): addr:40:44:00:00:00:02
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

We have 3 ports in the OVN bridge:

  1. ovn-centra-0: Tunnel port with Central node
  2. ovn-worker-0: Tunnel port with Worker1
  3. vm2: Our fake virtual machine that we bound to Worker2

When ICMP echo request packets are coming from VM1, we expect them to be arriving via port 2 and being delivered to VM2 on port 8. We can check this by inspecting the flows at table 0 (input) and table 65 (output) for br-int. For a more detailed information of the OpenFlow tables, see ovn-architecture(7) document, section "Architectural Physical Life Cycle of a Packet".

[root@worker2 ~]# ovs-ofctl dump-flows br-int table=0 | grep idle_age='[0-1],'
cookie=0x0, duration=3845.572s, table=0, n_packets=2979, n_bytes=291942, idle_age=0, priority=100,in_port=2 actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,33)


[root@worker2 ~]# ovs-ofctl dump-flows br-int table=65 | grep idle_age='[0-1],'
 cookie=0x0, duration=3887.336s, table=65, n_packets=3062, n_bytes=297780, idle_age=0, priority=100,reg15=0x1,metadata=0x8 actions=output:8

I was filtering the dump-flows output to display only those flows with an idle_age of just 0 or 1 seconds. This means that they've been recently hit so if we're inspecting tables with lots of flows, we would filter away unwanted flows to focus just on the ones that we're really interested. Please, note that for this particular example, I've set the VM2 link down to see only incoming ICMP packets from VM1 and hide the replies.

Let's inspect the input flow on table 0 where the following actions happen:

actions=
move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],
move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],
move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],
resubmit(,33)

These actions will:

  • Copy the 24 bits of the VNI (Tunnel ID / Logical Datapath) to the OpenFlow metadata.
  • Copy bits 16-30 of the tunnel data (Ingress port) to the OpenvSwitch register 14
  • Copy bits 0-15 of the tunnel data (Egress port) to the OpenvSwitch register 15.

From the ovn-architecture document, we can read:

logical datapath field
A field that denotes the logical datapath through which a
packet is being processed. OVN uses the field that Open‐
Flow 1.1+ simply (and confusingly) calls "metadata" to
store the logical datapath. (This field is passed across
tunnels as part of the tunnel key.)

logical input port field
A field that denotes the logical port from which the
packet entered the logical datapath. OVN stores this in
Open vSwitch extension register number 14.

Geneve and STT tunnels pass this field as part of the
tunnel key. Although VXLAN tunnels do not explicitly
carry a logical input port, OVN only uses VXLAN to commu‐
nicate with gateways that from OVN’s perspective consist
of only a single logical port, so that OVN can set the
logical input port field to this one on ingress to the
OVN logical pipeline.

logical output port field
A field that denotes the logical port from which the
packet will leave the logical datapath. This is initial‐
ized to 0 at the beginning of the logical ingress pipe‐
line. OVN stores this in Open vSwitch extension register
number 15.

Geneve and STT tunnels pass this field as part of the
tunnel key. VXLAN tunnels do not transmit the logical
output port field. Since VXLAN tunnels do not carry a
logical output port field in the tunnel key, when a
packet is received from VXLAN tunnel by an OVN hypervi‐
sor, the packet is resubmitted to table 8 to determine
the output port(s); when the packet reaches table 32,
these packets are resubmitted to table 33 for local de‐
livery by checking a MLF_RCV_FROM_VXLAN flag, which is
set when the packet arrives from a VXLAN tunnel. 

Similarly, we can see that in the output action from table 65, the OpenFlow rule is matching on "reg15=0x1,metadata=0x8" when it outputs the packet to the OpenFlow port number 8 (VM2). As we saw earlier, metadata 8 corresponds to network2 Logical Switch and the output port (reg15) 1 corresponds to the VM2 logical port (tunnel_key 1).

 

Due to its distributed nature and everything being OpenFlow based, it may be a little bit tricky to trace packets when implementing Software Defined Networking with OVN compared to traditional solutions (iptables, network namespaces, ...). Knowing the above concepts and getting familiar with the tools is key to debug OVN systems in an effective way.

OVN - Geneve Encapsulation

In the last post we created a Logical Switch with two ports residing on different hypervisors. Communication between those two ports took place over the tunnel interface using Geneve encapsulation. Let's now take a closer look at this overlay traffic.

Without diving too much into the packet processing in OVN, we need to know that each Logical Datapath (Logical Switch / Logical Router) has an ingress and an egress pipeline. Whenever a packet comes in, the ingress pipeline is executed and after the output action, the egress pipeline will run to deliver the packet to its destination. More info here: http://docs.openvswitch.org/en/latest/faq/ovn/#ovn

In our scenario, when we ping from VM1 to VM2, the ingress pipeline of each ICMP packet runs on Worker1 (where VM1 is bound to) and the packet is pushed to the tunnel interface to Worker2 (where VM2 resides). When Worker2 receives the packet on its physical interface, the egress pipeline of the Logical Switch (network1) is executed to deliver the packet to VM2. But ... How does OVN know where the packet comes from and which Logical Datapath should process it? This is where the metadata in the Geneve headers comes in.

Let's get back to our setup and ping from VM1 to VM2 and capture traffic on the physical interface (eth1) of Worker2:

[root@worker2 ~]# sudo tcpdump -i eth1 -vvvnnexx

17:02:13.403229 52:54:00:13:e0:a2 > 52:54:00:ac:67:5b, ethertype IPv4 (0x0800), length 156: (tos 0x0, ttl 64, id 63920, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.50.100.7549 > 192.168.50.101.6081: [bad udp cksum 0xe6a5 -> 0x7177!] Geneve, Flags [C], vni 0x1, proto TEB (0x6558), options [class Open Virtual Networking (OVN) (0x102) type 0x80(C) len 8 data 00010002]
        40:44:00:00:00:01 > 40:44:00:00:00:02, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 41968, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.0.11 > 192.168.0.12: ICMP echo request, id 1251, seq 6897, length 64
        0x0000:  5254 00ac 675b 5254 0013 e0a2 0800 4500
        0x0010:  008e f9b0 4000 4011 5a94 c0a8 3264 c0a8
        0x0020:  3265 1d7d 17c1 007a e6a5 0240 6558 0000
        0x0030:  0100 0102 8001 0001 0002 4044 0000 0002
        0x0040:  4044 0000 0001 0800 4500 0054 a3f0 4000
        0x0050:  4001 1551 c0a8 000b c0a8 000c 0800 c67b
        0x0060:  04e3 1af1 94d9 6e5c 0000 0000 41a7 0e00
        0x0070:  0000 0000 1011 1213 1415 1617 1819 1a1b
        0x0080:  1c1d 1e1f 2021 2223 2425 2627 2829 2a2b
        0x0090:  2c2d 2e2f 3031 3233 3435 3637

17:02:13.403268 52:54:00:ac:67:5b > 52:54:00:13:e0:a2, ethertype IPv4 (0x0800), length 156: (tos 0x0, ttl 64, id 46181, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.50.101.9683 > 192.168.50.100.6081: [bad udp cksum 0xe6a5 -> 0x6921!] Geneve, Flags [C], vni 0x1, proto TEB (0x6558), options [class Open Virtual Networking (OVN) (0x102) type 0x80(C) len 8 data 00020001]
        40:44:00:00:00:02 > 40:44:00:00:00:01, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16422, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.0.12 > 192.168.0.11: ICMP echo reply, id 1251, seq 6897, length 64
        0x0000:  5254 0013 e0a2 5254 00ac 675b 0800 4500
        0x0010:  008e b465 4000 4011 9fdf c0a8 3265 c0a8
        0x0020:  3264 25d3 17c1 007a e6a5 0240 6558 0000
        0x0030:  0100 0102 8001 0002 0001 4044 0000 0001
        0x0040:  4044 0000 0002 0800 4500 0054 4026 0000
        0x0050:  4001 b91b c0a8 000c c0a8 000b 0000 ce7b
        0x0060:  04e3 1af1 94d9 6e5c 0000 0000 41a7 0e00
        0x0070:  0000 0000 1011 1213 1415 1617 1819 1a1b
        0x0080:  1c1d 1e1f 2021 2223 2425 2627 2829 2a2b
        0x0090:  2c2d 2e2f 3031 3233 3435 3637

Let's now decode the ICMP request packet (I'm using this tool):

ICMP request inside the Geneve tunnel

Metadata

 

In the ovn-architecture(7) document, you can check how the Metadata is used in OVN in the Tunnel Encapsulations section. In short, OVN encodes the following information in the Geneve packets:

  • Logical Datapath (switch/router) identifier (24 bits) - Geneve VNI
  • Ingress and Egress port identifiers - Option with class 0x0102 and type 0x80 with 32 bits of data:
         1       15          16
       +---+------------+-----------+
       |rsv|ingress port|egress port|
       +---+------------+-----------+
         0

Back to our example: VNI = 0x000001 and Option Data = 00010002, so from the above:

Logical Datapath = 1   Ingress Port = 1   Egress Port = 2

Let's take a look at SB database contents to see if they match what we expect:

[root@central ~]# ovn-sbctl get Datapath_Binding network1 tunnel-key
1

[root@central ~]# ovn-sbctl get Port_Binding vm1 tunnel-key
1

[root@central ~]# ovn-sbctl get Port_Binding vm2 tunnel-key
2

We can see that the Logical Datapath belongs to network1, that the ingress port is vm1 and that the output port is vm2 which makes sense as we're analyzing the ICMP request from VM1 to VM2. 

By the time this packet hits Worker2 hypervisor, OVN has all the information to process the packet on the right pipeline and deliver the port to VM2 without having to run the ingress pipeline again.

What if we don't use any encapsulation?

This is technically possible in OVN and there's such scenarios like in the case where we're managing a physical network directly and won't use any kind of overlay technology. In this case, our ICMP request packet would've been pushed directly to the network and when Worker2 receives the packet, OVN needs to figure out (based on the IP/MAC addresses) which ingress pipeline to execute (twice, as it was also executed by Worker1) before it can go to the egress pipeline and deliver the packet to VM2.

Multinode OVN setup

As a follow up from the last post, we are now going to deploy a 3 nodes OVN setup to demonstrate basic L2 communication across different hypervisors. This is the physical topology and how services are distributed:

  • Central node: ovn-northd and ovsdb-servers (North and Southbound databases) as well as ovn-controller
  • Worker1 / Worker2: ovn-controller connected to Central node Southbound ovsdb-server (TCP port 6642)

In order to deploy the 3 machines, I'm using Vagrant + libvirt and you can checkout the Vagrant files and scripts used from this link. After running 'vagrant up', we'll have 3 nodes with OVS/OVN installed from sources and we should be able to log in to the central node and verify that OVN is up and running and Geneve tunnels have been established to both workers:

 

[vagrant@central ~]$ sudo ovs-vsctl show
f38658f5-4438-4917-8b51-3bb30146877a
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "ovn-worker-1"
            Interface "ovn-worker-1"
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.168.50.101"}
        Port "ovn-worker-0"
            Interface "ovn-worker-0"
                type: geneve
                options: {csum="true", key=flow, remote_ip="192.168.50.100"}
    ovs_version: "2.11.90"

 

For demonstration purposes, we're going to create a Logical Switch (network1) and two Logical Ports (vm1 and vm2). Then we're going to bind VM1 to Worker1 and VM2 to Worker2. If everything works as expected, we would be able to communicate both Logical Ports through the overlay network formed between both workers nodes.

We can run the following commands on any node to create the logical topology (please, note that if we run them on Worker1 or Worker2, we need to specify the NB database location by running ovn-nbctl with "--db=tcp:192.168.50.10:6641" as 6641 is the default port for NB database):

ovn-nbctl ls-add network1
ovn-nbctl lsp-add network1 vm1
ovn-nbctl lsp-add network1 vm2
ovn-nbctl lsp-set-addresses vm1 "40:44:00:00:00:01 192.168.0.11"
ovn-nbctl lsp-set-addresses vm2 "40:44:00:00:00:02 192.168.0.12"

And now let's check the Northbound and Southbound databases contents. As we didn't bind any port to the workers yet, "ovn-sbctl show" command should only list the chassis (or hosts in OVN jargon):

[root@central ~]# ovn-nbctl show
switch a51334e8-f77d-4d85-b01a-e547220eb3ff (network1)
    port vm2
        addresses: ["40:44:00:00:00:02 192.168.0.12"]
    port vm1
        addresses: ["40:44:00:00:00:01 192.168.0.11"]

[root@central ~]# ovn-sbctl show
Chassis "worker2"
    hostname: "worker2"
    Encap geneve
        ip: "192.168.50.101"
        options: {csum="true"}
Chassis central
    hostname: central
    Encap geneve
        ip: "127.0.0.1"
        options: {csum="true"}
Chassis "worker1"
    hostname: "worker1"
    Encap geneve
        ip: "192.168.50.100"
        options: {csum="true"}

Now we're going to bind VM1 to Worker1:

ovs-vsctl add-port br-int vm1 -- set Interface vm1 type=internal -- set Interface vm1 external_ids:iface-id=vm1
ip netns add vm1
ip link set vm1 netns vm1
ip netns exec vm1 ip link set vm1 address 40:44:00:00:00:01
ip netns exec vm1 ip addr add 192.168.0.11/24 dev vm1
ip netns exec vm1 ip link set vm1 up

And VM2 to Worker2:

ovs-vsctl add-port br-int vm2 -- set Interface vm2 type=internal -- set Interface vm2 external_ids:iface-id=vm2
ip netns add vm2
ip link set vm2 netns vm2
ip netns exec vm2 ip link set vm2 address 40:44:00:00:00:02
ip netns exec vm2 ip addr add 192.168.0.12/24 dev vm2
ip netns exec vm2 ip link set vm2 up

Checking again the Southbound database, we should see the port binding status:

[root@central ~]# ovn-sbctl show
Chassis "worker2"
    hostname: "worker2"
    Encap geneve
        ip: "192.168.50.101"
        options: {csum="true"}
    Port_Binding "vm2"
Chassis central
    hostname: central
    Encap geneve
        ip: "127.0.0.1"
        options: {csum="true"}
Chassis "worker1"
    hostname: "worker1"
    Encap geneve
        ip: "192.168.50.100"
        options: {csum="true"}
    Port_Binding "vm1"

Now let's check connectivity between VM1 (Worker1) and VM2 (Worker2):

[root@worker1 ~]# ip netns exec vm1 ping 192.168.0.12 -c2
PING 192.168.0.12 (192.168.0.12) 56(84) bytes of data.
64 bytes from 192.168.0.12: icmp_seq=1 ttl=64 time=0.416 ms
64 bytes from 192.168.0.12: icmp_seq=2 ttl=64 time=0.307 ms

--- 192.168.0.12 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.307/0.361/0.416/0.057 ms


[root@worker2 ~]# ip netns exec vm2 ping 192.168.0.11 -c2
PING 192.168.0.11 (192.168.0.11) 56(84) bytes of data.
64 bytes from 192.168.0.11: icmp_seq=1 ttl=64 time=0.825 ms
64 bytes from 192.168.0.11: icmp_seq=2 ttl=64 time=0.275 ms

--- 192.168.0.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.275/0.550/0.825/0.275 ms

As both ports are located in different hypervisors, OVN is pushing the traffic via the overlay Geneve tunnel from Worker1 to Worker2. In the next post, we'll analyze the Geneve encapsulation and how OVN uses its metadata internally.

For now, let's ping from VM1 to VM2 and just capture traffic on the geneve interface on Worker2 to verify that ICMP packets are coming through the tunnel:

[root@worker2 ~]# tcpdump -i genev_sys_6081 -vvnn icmp
tcpdump: listening on genev_sys_6081, link-type EN10MB (Ethernet), capture size 262144 bytes
15:07:42.395318 IP (tos 0x0, ttl 64, id 45147, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.0.11 > 192.168.0.12: ICMP echo request, id 1251, seq 26, length 64
15:07:42.395383 IP (tos 0x0, ttl 64, id 39282, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.0.12 > 192.168.0.11: ICMP echo reply, id 1251, seq 26, length 64
15:07:43.395221 IP (tos 0x0, ttl 64, id 45612, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.0.11 > 192.168.0.12: ICMP echo request, id 1251, seq 27, length 64

In coming posts we'll cover Geneve encapsulation as well as OVN pipelines and L3 connectivity.

Implementing Security Groups in OpenStack using OVN Port Groups

Some time back, when looking at the performance of OpenStack using OVN as the networking backend, we noticed that it didn't scale really well and it turned out that the major culprit was the way we implemented Neutron Security Groups . In order to illustrate the issue and the optimizations that we carried out, let's first explain how security was originally implemented:

Networking-ovn and Neutron Security Groups

Originally, Security Groups were implemented using a combination of OVN resources such as Address Sets and Access Control Lists (ACLs):

  • Address Sets: An OVN Address set contains a number of IP addresses that can be referenced from an ACL. In networking-ovn  we directly map Security Groups to OVN Address Sets: every time a new IP address is allocated for a port, this address will be added to the Address Set(s) representing the Security Groups which the port belongs to.
$ ovn-nbctl list address_set
_uuid : 039032e4-9d98-4368-8894-08e804e9ee78
addresses : ["10.0.0.118", "10.0.0.123", "10.0.0.138", "10.0.0.143"]
external_ids : {"neutron:security_group_id"="0509db24-4755-4321-bb6f-9a094962ec91"}
name : "as_ip4_0509db24_4755_4321_bb6f_9a094962ec91"
  • ACLs: They are applied to a Logical Switch (Neutron network). They have a 1-to-many relationship with Neutron Security Group Rules. For instance, when the user creates a single Neutron rule within a Security Group to allow ingress ICMP traffic, it will map to N ACLs in OVN Northbound database with N being the number of ports that belong to that Security Group.
$ openstack security group rule create --protocol icmp default
_uuid : 6f7635ff-99ae-498d-8700-eb634a16903b
action : allow-related
direction : to-lport
external_ids : {"neutron:lport"="95fb15a4-c638-42f2-9035-bee989d80603", "neutron:security_group_rule_id"="70bcb4ca-69d6-499f-bfcf-8f353742d3ff"}
log : false
match : "outport == \"95fb15a4-c638-42f2-9035-bee989d80603\" && ip4 && ip4.src == 0.0.0.0/0 && icmp4"
meter : []
name : []
priority : 1002
severity : []

On the other hand, Neutron has the possibility to filter traffic between ports within the same Security Group or a remote Security Group. One use case may be: a set of VMs whose ports belong to SG1 only allowing HTTP traffic from the outside and another set of VMs whose ports belong to SG2 blocking all incoming traffic. From Neutron, you can create a rule to allow database connections from SG1 to SG2. In this case, in OVN we'll see ACLs referencing the aforementioned Address Sets. In

$ openstack security group rule create --protocol tcp --dst-port 3306 --remote-group webservers default
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| created_at | 2018-12-21T11:29:32Z |
| description | |
| direction | ingress |
| ether_type | IPv4 |
| id | 663012c1-67de-45e1-a398-d15bd4f295bb |
| location | None |
| name | None |
| port_range_max | 3306 |
| port_range_min | 3306 |
| project_id | 471603b575184afc85c67d0c9e460e85 |
| protocol | tcp |
| remote_group_id | 11059b7d-725c-4740-8db8-5c5b89865d0f |
| remote_ip_prefix | None |
| revision_number | 0 |
| security_group_id | 0509db24-4755-4321-bb6f-9a094962ec91 |
| updated_at | 2018-12-21T11:29:32Z |
+-------------------+--------------------------------------+

This gets the following OVN ACL into Northbound database:


_uuid : 03dcbc0f-38b2-42da-8f20-25996044e516
action : allow-related
direction : to-lport
external_ids : {"neutron:lport"="7d6247b7-65b9-4864-a9a0-a85bacb4d9ac", "neutron:security_group_rule_id"="663012c1-67de-45e1-a398-d15bd4f295bb"}
log : false
match : "outport == \"7d6247b7-65b9-4864-a9a0-a85bacb4d9ac\" && ip4 && ip4.src == $as_ip4_11059b7d_725c_4740_8db8_5c5b89865d0f && tcp && tcp.dst == 3306"
meter : []
name : []
priority : 1002
severity : []

Problem "at scale"

In order to best illustrate the impact of the optimizations that the Port Groups feature brought in OpenStack, let's take a look at the number of ACLs on a typical setup when creating just 100 ports on a single network. All those ports will belong to a Security Group with the following rules:

  1. Allow incoming SSH traffic
  2. Allow incoming HTTP traffic
  3. Allow incoming ICMP traffic
  4. Allow all IPv4 traffic between ports of this same Security Group
  5. Allow all IPv6 traffic between ports of this same Security Group
  6. Allow all outgoing IPv4 traffic
  7. Allow all outgoing IPv6 traffic

Every time we create a port, new 10 ACLs (the 7 rules above + DHCP traffic ACL + default egress drop ACL + default ingress drop ACL) will be created in OVN:


$ ovn-nbctl list ACL| grep ce2ad98f-58cf-4b47-bd7c-38019f844b7b | grep match
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip6 && ip6.src == $as_ip6_0509db24_4755_4321_bb6f_9a094962ec91"
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip"
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4 && ip4.src == 0.0.0.0/0 && icmp4"
match : "inport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4"
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4 && ip4.src == $as_ip4_0509db24_4755_4321_bb6f_9a094962ec91"
match : "inport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip6"
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 80"
match : "outport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 22"
match : "inport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip4 && ip4.dst == {255.255.255.255, 10.0.0.0/8} && udp && udp.src == 68 && udp.dst == 67"
match : "inport == \"ce2ad98f-58cf-4b47-bd7c-38019f844b7b\" && ip"

With 100 ports, we'll observe 1K ACLs in the system:

$ ovn-nbctl lsp-list neutron-ebde771e-a93d-438d-a689-d02e9c91c7cf | wc -l
100
$ ovn-nbctl acl-list neutron-ebde771e-a93d-438d-a689-d02e9c91c7cf | wc -l
1000

When ovn-northd sees these new ACLs, it'll create the corresponding Logical Flows in Southbound database that will then be translated by ovn-controller to OpenFlow flows in the actual hypervisors. The number of Logical Flows also for this 100 ports system can be pulled like this:

$ ovn-sbctl lflow-list neutron-ebde771e-a93d-438d-a689-d02e9c91c7cf | wc -l
3052

At this point, you can pretty much tell that this doesn't look very promising at scale.

Optimization

One can quickly spot an optimization consisting on having just one ACL per Security Group Rule instead of one ACL per Security Group Rule per port if only we could reference a set of ports and not each port individually on the 'match' column of an ACL. This would alleviate calculations mainly on the networking-ovn side where we saw a bottleneck at scale when processing new ports due to the high number of ACLs.

Such optimization would require a few changes on the core OVN side:

  • Changes to the schema to create a new table in the Northbound database (Port_Group) and to be able to apply ACLs also to a Port Group.
  • Changes to ovn-northd so that it creates new Logical Flows based on ACLs applied to Port Groups.
  • Changes to ovn-controller so that it can figure out the physical flows to install on every hypervisor based on the new Logical Flows.

These changes happened mainly in the next 3 patches and the feature is present in OvS 2.10 and beyond:

https://github.com/openvswitch/ovs/commit/3d2848bafa93a2b483a4504c5de801454671dccf

https://github.com/openvswitch/ovs/commit/1beb60afd25a64f1779903b22b37ed3d9956d47c

https://github.com/openvswitch/ovs/commit/689829d53612a573f810271a01561f7b0948c8c8

On the networking-ovn side, we needed to adapt the code as well to:

  • Make use of the new feature and implement Security Groups using Port Groups.
  • Ensure a migration path from old implementation to Port Groups.
  • Keep backwards compatibility: in case an older version of OvS is used, we need to fall back to the previous implementation.

Here you can see the main patch to accomplish the changes above:

https://github.com/openstack/networking-ovn/commit/f01169b405bb5080a1bc1653f79512eb0664c35d

If we attempt to recreate the same scenario as we did earlier where we had 1000 ACLs for 100 ports on our Security Group using the new feature, we can compare the number of resources that we're now using:

$ ovn-nbctl lsp-list neutron-ebde771e-a93d-438d-a689-d02e9c91c7cf | wc -l
100

Two OVN Port Groups have been created: one for our Security Group and then neutron-pg-drop which is used to add fallback, low priority drop ACLs (by default OVN will allow all traffic if no explicit drop ACLs are added):

$ ovn-nbctl --bare --columns=name list Port_Group
neutron_pg_drop
pg_0509db24_4755_4321_bb6f_9a094962ec91

ACLs are now applied to Port Groups and not to the Logical Switch:

$ ovn-nbctl acl-list neutron-ebde771e-a93d-438d-a689-d02e9c91c7cf | wc -l
0
$ ovn-nbctl acl-list pg_0509db24_4755_4321_bb6f_9a094962ec91 | wc -l
7
$ ovn-nbctl acl-list neutron_pg_drop | wc -l
2

The number of ACLs has gone from 1000 (10 per port) to just 9 regardless of the number of ports in the system:

$ ovn-nbctl --bare --columns=match list ACL
inport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip4
inport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip6
inport == @neutron_pg_drop && ip
outport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 22
outport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip4 && ip4.src == 0.0.0.0/0 && icmp4
outport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip6 && ip6.src == $pg_0509db24_4755_4321_bb6f_9a094962ec91_ip6
outport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip4 && ip4.src == $pg_0509db24_4755_4321_bb6f_9a094962ec91_ip4
outport == @pg_0509db24_4755_4321_bb6f_9a094962ec91 && ip4 && ip4.src == 0.0.0.0/0 && tcp && tcp.dst == 80
outport == @neutron_pg_drop && ip

 

This change was merged in OpenStack Queens and requires OvS 2.10 at least. Also, if upgrading from an earlier version of either OpenStack or OvS, networking-ovn will take care of the migration from Address Sets to Port Groups upon start of Neutron server and the new implementation will be automatically used.

As  a bonus, this enables the possibility of applying the conjunctive match action easier on Logical Flows resulting in a big performance improvement as it was reported here.

Encrypting your connections with stunnel

stunnel is an open source software that provides SSL/TLS tunneling. This is especially useful when it comes to protect existing client-server communications that do not provide any encryption at all. Another application is to avoid exposing many services and make all of them pass through the tunnel and, therefore, securing all the traffic at the same time.

And because I have a WR703N with an OpenVPN server installed, I decided to set up stunnel and give it a try. The advantage over using my existing VPN, under certain circumstances, is that the establishment of the secure tunnel looks pretty much like a normal connection to an HTTPS website so most of the networks/proxys will allow this traffic whilst the VPN might be blocked (especially if UDP is used). So, the OpenVPN+stunnel combo looks like a pretty good security solution to be installed on our OpenWRT device.

The way I have the stunnel service configured is using MTLS (client and server authentication) and allowing only TLSv1.2 protocol. These are the specific lines in the stunnel.conf (server side):

; protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = all
options = CIPHER_SERVER_PREFERENCE
options = NO_SSLv2
options = NO_SSLv3
options = NO_TLSv1

Just for testing, I have installed stunnel on a Windows box and configured it as a client (with a client certificate signed by the same CA as the server) and connections to server port 443 will be forwarded to the SSH service running on the server side. This would allow us to SSH our server without
needing to expose it and, for example, set up a SOCKS proxy and browse the internet securely through the tunnel.

stunnel Diagram

Client side:

[https]
accept  = 22
protocol = connect
connect = proxy:8080
protocolHost= server:443

Server side:

[https]
accept  = 443
connect = 22
TIMEOUTclose = 0

On the client side, simply SSH localhost on the configured port (22) and stunnel will intercept this connection and establish a TLS tunnel with the server to the SSH service running on it.

These are the logs on the client side when SSH'ing localhost:

2016.07.20 21:37:09 LOG7[12]: Service [https] started
2016.07.20 21:37:09 LOG5[12]: Service [https] accepted connection from 127.0.0.1:43858
2016.07.20 21:37:09 LOG6[12]: s_connect: connecting proxy:8080
2016.07.20 21:37:09 LOG7[12]: s_connect: s_poll_wait proxy:8080: waiting 10 seconds
2016.07.20 21:37:09 LOG5[12]: s_connect: connected proxy:8080
2016.07.20 21:37:09 LOG5[12]: Service [https] connected remote server from x.x.x.x:43859
2016.07.20 21:37:09 LOG7[12]: Remote descriptor (FD=732) initialized
2016.07.20 21:37:09 LOG7[12]:  -> CONNECT server:443 HTTP/1.1
2016.07.20 21:37:09 LOG7[12]:  -> Host: server:443
2016.07.20 21:37:09 LOG7[12]:  -> Proxy-Authorization: basic **
2016.07.20 21:37:09 LOG7[12]:  -> 
2016.07.20 21:37:09 LOG7[12]:  <- HTTP/1.1 200 Connection established
2016.07.20 21:37:09 LOG6[12]: CONNECT request accepted
2016.07.20 21:37:09 LOG7[12]:  <- 
2016.07.20 21:37:09 LOG6[12]: SNI: sending servername: server
2016.07.20 21:37:09 LOG7[12]: SSL state (connect): before/connect initialization
2016.07.20 21:37:09 LOG7[12]: SSL state (connect): SSLv3 write client hello A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server hello A
2016.07.20 21:37:11 LOG7[12]: Verification started at depth=1: C=ES, ST=M, O=O, CN=wrtServer
2016.07.20 21:37:11 LOG7[12]: CERT: Pre-verification succeeded
2016.07.20 21:37:11 LOG6[12]: Certificate accepted at depth=1: C=ES, ST=M, O=O, CN=wrtServer
2016.07.20 21:37:11 LOG7[12]: Verification started at depth=0: C=ES, ST=S, O=O, CN=wrtClient
2016.07.20 21:37:11 LOG7[12]: CERT: Pre-verification succeeded
2016.07.20 21:37:11 LOG5[12]: Certificate accepted at depth=0: C=ES, ST=S, O=O, CN=wrtClient
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server certificate A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server key exchange A
2016.07.20 21:37:11 LOG6[12]: Client CA: C=ES, ST=M, O=O, CN=wrtCA
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server certificate request A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server done A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write client certificate A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write client key exchange A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write certificate verify A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write change cipher spec A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write finished A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 flush data
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server session ticket A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read finished A
2016.07.20 21:37:11 LOG7[12]:      8 client connect(s) requested
2016.07.20 21:37:11 LOG7[12]:      7 client connect(s) succeeded
2016.07.20 21:37:11 LOG7[12]:      0 client renegotiation(s) requested
2016.07.20 21:37:11 LOG7[12]:      2 session reuse(s)
2016.07.20 21:37:11 LOG6[12]: SSL connected: new session negotiated
2016.07.20 21:37:11 LOG7[12]: Deallocating application specific data for addr index
2016.07.20 21:37:11 LOG6[12]: Negotiated TLSv1.2 ciphersuite ECDHE-RSA-AES256-GCM-SHA384 (256-bit encryption)

As you can see, the traffic will be routed through a TLSv1.2 channel encrypted with AES256 in GCM mode and the session key has been derived using ephimeral ECDH, with Perfect Forward Secrecy so the traffic will be fairly well protected, at least, up to the stunnel server.

Make sure to keep an eye on the vulnerabilities listed on the stunnel website and have the server properly patched.

Building RPM packages

I wanted to learn how to build an RPM package out of a Python module so, now that I'm playing a bit with OpenStack, I decided to pick up a log merger for OpenStack files and build the corresponding package on my Fedora 24.

First thing is to setup the distribution with the right packages:

[root@localhost ~]$ dnf install @development-tools fedora-packager
[dani@localhost ~]$ rpmdev-setuptree
[dani@localhost ~]$ ls rpmbuild/
BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS

Now, under the SPECS directory, we need to create the spec file which will include the necessary info to build the RPM:

%global srcname os-log-merger
%global	sum	OpenStack Log Merger

Name:		python-%{srcname}
Version:	1.0.6
Release:	1%{?dist}
Summary:	%{sum}

License:	Apache
URL:		https://github.com/mangelajo/os-log-merger/
Source:         https://pypi.python.org/packages/6f/f1/b2a46907086c29725dd0e2296d6f45e7965670a05b43626abc1c81a098a0/os-log-merger-%{version}.tar.gz

BuildRoot:      %{_tmppath}/%{srcname}-%{version}-build
BuildArch:	noarch
BuildRequires:	python2

%description
A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

%package -n %{srcname}
Summary:	%{sum}
%{?python_provide:%python_provide python2-%{srcname}}

%description -n %{srcname}
A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

%prep
%autosetup -n %{srcname}-%{version}

%install
%py2_install

%check
%{__python2} setup.py test

%files -n %{srcname}
#%license LICENSE
%doc README.rst
%{python2_sitelib}/*
%{_bindir}/os-log-merger
%{_bindir}/oslogmerger
%{_bindir}/netprobe

%changelog
* Tue Jul 19 2016 dani - 1.0.6-1
- First version of the os-log-merger-package

Once the file is created, it's time to build the RPM package:

[dani@localhost SPECS]$ rpmbuild -bb os-log-merger.spec 
....
+ umask 022
+ cd /home/dani/rpmbuild/BUILD
+ cd os-log-merger-1.0.6
+ /usr/bin/rm -rf /home/dani/rpmbuild/BUILDROOT/python-os-log-merger-1.0.6-1.fc24.x86_64
+ exit 0
[dani@localhost SPECS]$ ls -alh ../RPMS/noarch/
total 44K
drwxr-xr-x. 2 dani dani 4,0K jul 19 20:35 .
drwxr-xr-x. 3 dani dani 4,0K jul 19 20:35 ..
-rw-rw-r--. 1 dani dani  34K jul 19 20:47 os-log-merger-1.0.6-1.fc24.noarch.rpm

We can see that the rpmbuild command produced the RPM file inside ~/rpmbuild/RPMS/noarch. Let's pull the info from it and check whether it's correct:

[dani@localhost SPECS]$ rpm -qip ../RPMS/noarch/os-log-merger-1.0.6-1.fc24.noarch.rpm 
Name        : os-log-merger
Version     : 1.0.6
Release     : 1.fc24
Architecture: noarch
Install Date: (not installed)
Group       : Unspecified
Size        : 85356
License     : Apache
Signature   : (none)
Source RPM  : python-os-log-merger-1.0.6-1.fc24.src.rpm
Build Date  : mar 19 jul 2016 20:47:42 CEST
Build Host  : localhost
Relocations : (not relocatable)
URL         : https://github.com/mangelajo/os-log-merger/
Summary     : OpenStack Log Merger
Description :
A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

The last step is trying to install the actual file and execute the module to see if everything went fine:

[root@localhost noarch]$ rpm -qa | grep os-log-merger
[root@localhost noarch]$ rpm -i os-log-merger-1.0.6-1.fc24.noarch.rpm 
[root@localhost noarch]$ oslogmerger 
usage: oslogmerger [-h] [-v] [--log-base  LOG_BASE]
                   [--log-postfix  LOG_POSTFIX] [--alias-level ALIAS_LEVEL]
                   [--min-memory] [--msg-logs file[:ALIAS] [file[:ALIAS] ...]]
                   [--timestamp-logs file[:ALIAS] [file[:ALIAS] ...]]
                   log_file[:ALIAS] [log_file[:ALIAS] ...]

References:
https://fedoraproject.org/wiki/How_to_create_a_GNU_Hello_RPM_package
https://fedoraproject.org/wiki/Packaging:Python

OpenVPN & OpenWRT - Secure Browsing from your mobile phone

Some time ago, I bought a TL-WR703N WiFi router for less than 15€. It came with a Chinese firmware that I overwrote with an OpenWRT image and connected to my ADSL router.

This device is really cool but once you flash it with an OpenWRT image you'll find out that there's almost no free space  (4MB total flash) so I decided to use an external USB memory to increase the available space and turn it into a useful gadget 🙂
WR703N

 

I used to have stunnel and OpenVPN servers running on my PC but since I didn't want to have the computer on all day, I decided to replace it with this small device which makes no noise and consumes very low power (around 80mA/~0.4W with WiFi on if I disable the blue LED :))

First thing I did was setting up an AP and disable my router's WiFi network since its antenna was surprisingly better and my old router doesn't support WiFi n. So my devices at home would connect to the WR703N WiFi network which was bridged with the ethernet interface to the ADSL router.

After this introduction, I'll explain how to set up an OpenVPN server to browse securely anywhere from your phone which is especially useful if you're using free or untrusted wifi networks out there.

 

OpenVPN Diagram

OpenVPN Diagram

At the moment of writing, I'm on the latest OpenWRT version which is Attitude Adjustment 12.09, r36088.

Required packages:

  • openvpn-easy-rsa - 2.2.2-2 - Simple shell scripts to manage a Certificate Authority
  • openvpn - 2.2.2-2 - Open source VPN solution using SSL

 

1. Certificates and keys generation

The easy-rsa package helps you create the CA, server and client certificates but you can either create them yourself or use existing ones created somehow (as long as you keep your private keys secret 🙂

I created 2048 bit RSA certificates with the help of the easy-rsa tool:

Edit the /etc/easy-rsa/vars file and change the KEY_SIZE value to 2048

export KEY_SIZE=2048

Also, feel free to change the certificate public data such as the common name, country, etc.

 

root@OpenWrt:/etc/easy-rsa# build-ca
NOTE: If you run ./clean-all, I will be doing a rm -rf on /etc/easy-rsa/keys
Generating a 2048 bit RSA private key
..................................+++
writing new private key to 'ca.key'
-----

Now generate the DH parameters and the server and client certificates signed by the previous CA.

build-dh
build-key-server server
build-key-pkcs12 daniiphone

Afterwards, all those files will be located under /etc/easy-rsa/keys and must be copied over to the openvpn directory:
root@OpenWrt:/etc/easy-rsa/keys# cp ca.crt ca.key dh2048.pem server.crt server.key /etc/openvpn/

The client certificate is not needed on the server side but we've generated it inside the WR703N and must be copied onto the client (in this case, my iPhone). We've generated the client certificate (daniiphone) in PKCS12 format which includes both the public certificate and the private key. It's really important to protect it with a password because you're gonna send it to your e-mail in order to import it from the OpenVPN iOS app. At the time of creating it, you'll be prompted to enter a password.

 

2.  Server side configuration

I've configured the OpenVPN server as follows using uci (you can do it by editing /etc/config/openvpn file).

root@OpenWrt:/etc/openvpn# uci show openvpn
openvpn.myvpn=openvpn
openvpn.myvpn.enable=1
openvpn.myvpn.port=1194
openvpn.myvpn.proto=udp
openvpn.myvpn.dev=tun
openvpn.myvpn.ca=/etc/openvpn/ca.crt
openvpn.myvpn.cert=/etc/openvpn/server.crt
openvpn.myvpn.key=/etc/openvpn/server.key
openvpn.myvpn.dh=/etc/openvpn/dh2048.pem
openvpn.myvpn.ifconfig_pool_persist=/tmp/ipp.txt
openvpn.myvpn.keepalive=10 120
openvpn.myvpn.persist_key=1
openvpn.myvpn.persist_tun=1
openvpn.myvpn.status=/var/log/openvpn-status.log
openvpn.myvpn.verb=3
openvpn.myvpn.server=10.8.0.0 255.255.255.0
openvpn.myvpn.client_to_client=1
openvpn.myvpn.comp_lzo=1
openvpn.myvpn.push=route 192.168.1.0 255.255.255.0 dhcp-option DNS 192.168.1.1 dhcp-option DOMAIN 192.168.1.1

These settings tell the server to listen on UDP port 1194 (which needs to be forwarded in your ADSL router to the WR703N IP address) and sets the VPN network at 10.8.0.0/24 (clients will be assigned an IP address in this subnet).
The last line creates a default route to my lan network 192.168.1.0/24 and shall be replaced with your own configuration.

Now we need to create a rule in the firewall to permit the VPN traffic. Add the following rule to the /etc/config/firewall file on the OpenWRT system:
config 'rule'
option 'target' 'ACCEPT'
option 'name' 'vpn'
option 'src' 'wan'
option 'proto' 'udp'
option 'dest_port' '1194'

In order to forward traffic from the VPN to the wan connection, we need to enable forwarding on the tun interface and create an NAT to the local interface:

iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT
iptables -I OUTPUT -o tun+ -j ACCEPT
iptables -I FORWARD -o tun+ -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source 192.168.1.5

Replace the 192.168.1.5 with your OpenWRT lan IP address and start the openvpn service:

Enable OpenVPN service autostart
root@OpenWrt:~# /etc/init.d/openvpn enable
Start the service
root@OpenWrt:~# /etc/init.d/openvpn start

 

2.  Client side configuration

Now we need to setup the iPhone (it should work on Android phones or any computer running OpenVPN) to connect to our VPN server. First we need to transfer the .p12 file created in the first step and import it to the phone from an e-mail attachment.

Install Certificate

Install Certificate

Certificate Installed

Certificate Installed


Once you have successfully imported the certificate file to the iPhone, it's time to load the OpenVPN configuration. In order to load it easily, create a .OVPN file like this one:

client
dev tun
proto udp
remote your-public-ip-address 1194
comp-lzo
redirect-gateway
<ca>
-----BEGIN CERTIFICATE-----
MIIE3zCCA8egAwIB/
....
oZEG
-----END CERTIFICATE-----
</ca>
nobind
persist-key
persist-tun
user nobody
group nogroup
resolv-restry infinite

Make sure you specify your public IP address and your CA certificate inside the configuration. This CA is needed because we have used a self-signed CA which is not trusted by the OS so if we didn't include this certificate within the configuration, the OpenVPN client would not trust the certificate presented by the server during the TLS negotiation.

This .OVPN file has to be imported from an e-mail attachment directly into the OpenVNP app. Once imported, click on the green "Add" button to associate the previous certificate and its private key to this profile. In your configuration you should be able to see your remote IP address (or hostname) instead of "80.80." which I've edited in the screenshots below.

Import OVPN file

Import OVPN file

Add imported certificate

Add imported certificate

 

OpenVPN Profile

OpenVPN Profile

OpenVPN connected

OpenVPN connected

Now that the tunnel has been setup, you should be able to see the "VPN" symbol on the status bar of your iPhone and all your traffic will be encrypted up to your home network. In order to test the connectivity and the forwarding rules, I try to access the OpenWRT Luci web configuration by typing the WR703N IP address in Safari:

OpenWRT Luci

OpenWRT Luci

Now, your iPhone is connected to your home network and all the traffic will go through your ADSL connection. Anyone trying to eavesdrop on the WiFi network will only be able to see tons of encrypted traffic.

It's very important that you keep the CA private key secret in order to avoid "man-in-the-middle" attacks, as well as protect the .p12 file when you send it over to your phone.

L8 SmartLight - Our Kickstarter adventure!

It’s been quite a long time since I don’t update this blog. The reason why is that I’ve been too busy with L8 Smartlight, a new entrepreneurship project that some friends and I have started a few months ago.

We’re very proud of being the first Spanish project to be successful on Kickstarter and, now that the funding project is about to finish, we’re very keen on starting the production of this new gadget and looking forward to spreading the world with it!

So far, the project’s had sparked the interest of many important sites such as Mashable:

We’re currently working on fully supporting Bluetooth EDR and 4.0 to cover most of the smartphones in the Market. Also there’s still a long way to go to define all the APIs and functionality that will be hopefully ready in the next few months.

I’ll try to update the status on the L8 project in this blog but, we’ll make sure that the latest information will be available right away at Kickstarter

Dani

SecuDroid - Android Anti Theft Loss app Review

SecuDroid - Anti Theft/Loss app for Android

Google is now activating over 500,000 devices each day, growing at 4.4% week on week. This huge raise, led to a big explosion of apps being downloaded from the Android Market, which - at the time of this writing - is 6,148,593,955 (more info at http://www.androlib.com/appstats.aspx).

Among these, there is one particular kind of application which is potentially useful for every single Android user: an app which allows you to locate, track or find your phone in case of theft or loss. SecuDroid is one of those must-have apps which you'd better not need to use, but you'll definitely be glad to have it when somebody steals your phone or you lose it somewhere.

 Features: 

  • Location information including accuracy, speed and altitude 
  • Periodic tracking
  • Silent pictures using either the front or back camera
  • Remote Lock & Wipe functions
  • SIM card change notification 
  • FindMe Ringtone
  • Invisible Mode
  • E-mail integration  

FAQ: 

  • How does SecuDroid work?

SecuDroid sleeps silently in your Android device waiting for commands sent from any cellphone by SMS. The answer to these comands will be sent by SecuDroid either by e-mail or by SMS.

These SMS will not show up on your phone because they are intercepted by SecuDroid, so the thief will not suspect that you are on your way to recover your phone.

  • How can I be sure that nobody will be able to track me if I install SecuDroid?

When you request an action from SecuDroid you must precede the command with a password. This password can be changed at any times from the SecuDroid configuration window and, by default, it is set to "PW" (without quotes).  Anyone who knows this password will be able to track you or even wipe your data off your phone.

  • How can I hide SecuDroid so that noone knows that it's there?

SecuDroid will appear as "eNotes" under the installed applications list. Also, you will be able to remove the launcher icon from the configuration window.

  • If I remove the launcher icon, how can I modify the settings?

You'll be able to launch the configuration window by dialing a pre-configured code from the stock phone app. This code, by default, is 3535 but you can change it at any times.

  • Will SecuDroid be able to determine my location even if GPS is off?

In order to switch GPS on, Android enforces the user to do it manually. However, SecuDroid will enable the GPS hardware without user interaction (by tricking the OS) to get the location and, afterwards, it will switch it off automatically to hide itself. Nonetheless, it is highly recommended that you test this feature in your particular device because this is likely to be avoided by Android in the future. In this case, just leave the GPS setting on (no impact on battery life) or you'll only get positions based on GSM towers triangulation.

  • Somebody stole my phone and I want to find out who, will the thief be able to locate the pictures in the phone storage?

No. Pictures are stored temporally without image extensions and they will be erased right after being sent

  • If I take a picture remotelly, will the thief hear the camera shutter sound?

SecuDroid will take pictures silently by disabling the shutter sound. However, some devices are impossible to get the shutter sound disabled due to legal restrictions so you must make sure how your device works by testing it first. You'll probably be able to disable it by some other means like rooting your phone and removing the sound but SecuDroid doesn't encourage you to root your phone unlike other apps.

  • The FindMe function is pretty cool but what if I left the phone in silent/vibration mode?

Just send a BEEP command to your phone and SecuDroid will ring loud for 60 seconds or until you switch the screen on/off no matter what the previous state was. After this command, volume will be set at maximum leve so you're free now to call and find out where the sound comes from

  • Just realized that I lost my phone with critical information. How can SecuDroid help me?

Before trying to track or locate your phone, you should lock it with a password to make sure that noone can access to your sensitive data or even make calls. If you feel that the information inside your phone is even more important than the device itself, wipe the data within a few seconds by issuing a WIPE command.

  • Transparency is really important. Does SecuDroid include any phone-home component that could compromise my privacy?

SecuDroid does NOT include any phone-home component. You can be completely sure that SecuDroid will not connect to a 3rd party server and will not use your location or personal information in any ways.

  • What e-mail account does SecuDroid use to send the pictures and location info to?

You can configure a GMail account from SecuDroid settings window and this will be the account used to send the answers to remote requests. If, for example, this account is removed or you change the password, the answers will be sent back via SMS (or MMS if a picture was requested). The default destination e-mail address is also configured and has not necessarily to be a GMail address.

Details: http://www.secudroidapp.com/secudroid
Instructions: http://www.secudroidapp.com/instructions
FAQ: http://www.secudroidapp.com/faq

Silvestre & uXbot - Line Following Champions

Last week, we attended two of the most important championships in Spain with Silvestre and uXbot.

SilvestreuXbot

  • In Hispabot, the track had a ramp which was pretty challenging for our robots since they were not designed with this in mind. The major problem was that the robots got blind for a while in their way up and also in their way down. This problem was worse for Silvestre since it's larger and its sensors spent more time in the air, reason why the uXbot managed to get to the top of the podium with Silvestre right behind him on the 2nd place.
  • Robolid hold a really crowded championship with around 80 robots taking part in the robotrackers category and 20 robots in the Line Following contest. Silvestre and uXbot had to fight with most of the best robots in Spain and they finally won with Silvestre in the 1st place and uXbot 2nd.  On a side note, a new electronic system was used during the contest to measure lap times, average and top speeds and Silvestre's top speed was over 5 meters per second (18 km/h)!

The most remarkable features of the robots are:

  • DC Maxon motors
  • FPGA to accurately read the quadrature signals from the encoders and real-time processing (Silvestre)
  • Inertial sensors
  • Track learning: the algorithm takes about 2-2.5 laps to identify when the lap starts over and extract the characteristics and speed-up/braking points.
  • Adaptative behaviour: Throughout the laps, the robots modify their parameters (speed, speed-up/braking points,...) for each part of the track. This allows the robot to adapt themselves to the track conditions (dust, creases, ...)

Below you can see a video from the training days to the final round of the latest contest. Some slow motion parts show how the robots performed over the ramp at Hispabot and how they had to fight with the skids right after the downhill 😉

D.