OVN - Profiling and optimizing ports creation

One of the most important abstractions to handle virtual networking in OpenStack clouds are definitely ports. A port is basically a connection point for attaching a single device, such as the NIC of a server to a network, a subnet to a router, etc.  They represent entry and exit points for data traffic playing a critical role in OpenStack networking.

Given the importance of ports, it's clear that any operation on them should perform well, specially at scale and we should be able to catch any regressions ASAP before they land on production environments. Such tests should be done in a periodic fashion and should require doing them on a consistent hardware with enough resources.

In one of our first performance tests using OVN as a backend for OpenStack networking, we found out that port creation would be around 20-30% slower than using ML2/OVS. At this point, those are merely DB operations so there had to be something really odd going on.  First thing I did was to measure the time for a single port creation by creating 800 ports in an empty cloud. Each port would belong to a security group with allowed egress traffic and allowed ingress ICMP and SSH. These are the initial results:

As you can see, the time for creating a single port grows linearly with the number of ports in the cloud.  This is obviously a problem at scale that requires further investigation.

In order to understand how a port gets created in OVN, it's recommended to take a look at the NorthBound database schema and to this interesting blogpost by my colleague Russel Bryant about how Neutron security groups are implemented in OVN. When a port is first created in OVN, the following DB operations will occur:

  1.  Logical_Switch_Port 'insert'
  2.  Address_Set 'modify'
  3.  ACL 'insert' x8 (one per ACL)
  4.  Logical_Switch_Port 'modify' (to add the new 8 ACLs)
  5.  Logical_Switch_Port 'modify' ('up' = False)

After a closer look to OVN NB database contents, with 800 ports created, there'll be a total of 6400 ACLs (8 per port) which look all the same except for the inport/outport fields. A first optimization looks obvious: let's try to get all those 800 ports into the same group and make the ACLs reference that group avoiding duplication. There's some details and discussions in OpenvSwitch mailing list here. So far so good; we'll be cutting down the number of ACLs to 8 in this case no matter how big the number of ports in the system increases. This would reduce the number of DB operations considerably and scale better.

However, I went deeper and wanted to understand this linear growth through profiling. I'd like to write a separate blogpost about how to do the profiling but, so far, what I found is that the 'modify' operations on the Logical_Switch table to insert the new 8 ACLs would take longer and longer every time (the same goes to the Address_Set table where a new element is added to it with every new port). As the number of elements grow in those two tables, inserting a new element was more expensive. Digging further in the profiling results, it looks like "__apply_diff()" method is the main culprit, and more precisely, "Datum.to_json()". Here's a link to the actual source code.

Whenever a 'modify' operation happens in OVSDB, ovsdb-server will send a JSON  update notification to all clients (it's actually an update2 notification which is essentially the same but including only the diff with the previous state). This notification is going to be processed by the client (in this case networking-ovn, the OpenStack integration) in the "process_update2" method here and it happens that the Datum.to_json() method is taking most of the execution time. The main questions that came to my mind at this point are:

  1. How is this JSON data later being used?
  2. Why do we need to convert to JSON if the original notification was already in JSON?

For 1, it looks like the JSON data is used to build a Row object which will be then used to notify networking-ovn of the change just in case it'd be monitoring any events:

    def __apply_diff(self, table, row, row_diff):
        old_row_diff_json = {}
        for column_name, datum_diff_json in six.iteritems(row_diff):
            old_row_diff_json[column_name] = row._data[column_name].to_json()
        return old_row_diff_json
old_row_diff_json = self.__apply_diff(table, row,
self.notify(ROW_UPDATE, row,
            Row.from_json(self, table, uuid, old_row_diff_json))

Doesn't it look pointless to convert data "to_json" and then back "from_json"? As the number of elements in the row grows, it'll take longer to produce the associated JSON document (linear growth) of the whole column contents (not just the diff) and the same would go the other way around.

I thought of finding a way to build the Row from the data itself before going through the expensive (and useless) JSON conversions. The patch would look something like this:

diff --git a/python/ovs/db/ b/python/ovs/db/
index 60548bcf5..5a4d129c0 100644
--- a/python/ovs/db/
+++ b/python/ovs/db/
@@ -518,10 +518,8 @@ class Idl(object):
             if not row:
                 raise error.Error('Modify non-existing row')
-            old_row_diff_json = self.__apply_diff(table, row,
-                                                  row_update['modify'])
-            self.notify(ROW_UPDATE, row,
-                        Row.from_json(self, table, uuid, old_row_diff_json))
+            old_row = self.__apply_diff(table, row, row_update['modify'])
+            self.notify(ROW_UPDATE, row, Row(self, table, uuid, old_row))
             changed = True
             raise error.Error('<row-update> unknown operation',
@@ -584,7 +582,7 @@ class Idl(object):
                         row_update[] = self.__column_name(column)
     def __apply_diff(self, table, row, row_diff):
-        old_row_diff_json = {}
+        old_row = {}
         for column_name, datum_diff_json in six.iteritems(row_diff):
             column = table.columns.get(column_name)
             if not column:
@@ -601,12 +599,12 @@ class Idl(object):
                           % (column_name,, e))
-            old_row_diff_json[column_name] = row._data[column_name].to_json()
+            old_row[column_name] = row._data[column_name].copy()
             datum = row._data[column_name].diff(datum_diff)
             if datum != row._data[column_name]:
                 row._data[column_name] = datum
-        return old_row_diff_json
+        return old_row
     def __row_update(self, table, row, row_json):
         changed = False

Now that we have everything in place, it's time to repeat the tests with the new code and compare the results:

The improvement is obvious! Now the time to create a port is mostly constant regardless of the number of ports in the cloud and the best part is that this gain is not specific to port creation but to ANY modify operation taking place in OVSDB that uses the Python implementation of the OpenvSwitch IDL.

The intent of this blogpost is to show how to deal with performance bottlenecks through profiling and debugging in a top-down fashion, first through simple API calls measuring the time they take to complete and then all the way down to the database changes notifications.


OpenStack: Deploying a new containerized service in TripleO

When I first started to work in networking-ovn one of the first tasks I took up was to implement the ability for instances to fetch userdata and metadata at boot, such as the name of the instance, public keys, etc.

This involved introducing a new service which we called networking-ovn-metadata-agent and it's basically a process running in compute nodes that intercepts requests from instances within network namespaces, adds some headers and forwards those to Nova. While it was fun working on it I soon realized that most of the work would be on the TripleO side and, since I was really new to it, I decided to take the challenge as well!
If you're interested in the actual code for this feature (not the deployment related code), I sent the following patches to implement it and I plan to write a different blogpost for it:

But implementing this feature didn't end there and we had to support a way to deploy the new service from TripleO and, YES!, it has to be containerized. I found out that this was not a simple task so I decided to write this post with the steps I took hoping it helps people going through the same process. Before describing those, I want to highlight a few things/tips:

  • It's highly recommended to have access to a hardware capable of deploying TripleO and not relying only on the gate. This will speed up the development process *a lot* and lower pressure on the gate, which sometimes has really long queues.
  • The jobs running on the upstream CI use one node for the undercloud and one node for the overcloud so it's not always easy to catch failures when you deploy services on certain roles like in this case where we only want the new service in computes.  CI was green on certain patchsets while I encountered problems on 3 controllers + 3 computes setups due to this. Usually production environments would be HA so it's best to do the development on a more realistic setups whenever possible.
  • As of today, with containers, there's no way that you can send a patch on your project (in this case networking-ovn), add a Depends-On in tripleo-* and expect that the new change is tested. Instead, the patch has to get merged, an RDO promotion has to occur so that the RPM with the fix is available and then, the container images get built and ready to be fetched by TripleO jobs. This is a regression when compared to non-containerized jobs and clearly slows down the development process. TripleO folks are doing a great effort to include a mechanism that supports this which will be a huge leap 🙂
  • To overcome the above, I had to go over the process of building my own kolla images and setting them up in the local registry (more info here). This way you can do your own testing without having to wait for the next RDO promotion. Still, your patches won't be able to merge until it happens but you can keep on developing your stuff.
  • To test any puppet changes, I had to patch the overcloud image (usually mounting it with guestmount and changing the required files) and update it before redeploying. This is handy as well as the 'virt-customize' command  which allows you to execute commands directly in the overcloud image, for example, to install a new RPM package (for me usually upgrading OpenvSwitch for testing). This is no different from baremetal deployment but still useful here.

After this long introduction, let me go through the actual code that implements this new service.


1. Getting the package and the container image ready:

At this point we should be able to consume the container image from TripleO. The next step is to configure the new service with the right parameters.

2. New service configuration:


This new service will require some configuration options. Writing those configuration options will be done by puppet and, since it is a networking service, I sent a patch to puppet-neutron to support it:

All the configuration options as well as the service definition are in this file:

As we also want to set a knob which enables/disables the service in the ML2 plugin, I added an option for that here:

The patch also includes unit tests and a release note as we're introducing a new service.


We want this service to be deployed at step 4 (see OpenStack deployment steps for more information) and also, we want networking-ovn-metadata-agent to be started after ovn-controller is up and running. ovn-controller service will be started after OpenvSwitch so this way we'll ensure that our service will start at the right moment. For this, I sent a patch to puppet-tripleo:

And later, I found out that I wanted the neutron base profile configuration to be applied to my own service. This way I could benefit from some Neutron common configuration such as logging files, etc.:

3. Actual deployment in tripleo-heat-templates:

This is the high-level work that drives all the above and, initially, I had it in three different patches which I ended up squashing because of the inter-dependencies.

To sum up, this is what this patch does

I hope this help others introducing new services in OpenStack! Feel free to reach out to me for any comments/corrections/suggestions by e-mail or on IRC 🙂

Encrypting your connections with stunnel

stunnel is an open source software that provides SSL/TLS tunneling. This is especially useful when it comes to protect existing client-server communications that do not provide any encryption at all. Another application is to avoid exposing many services and make all of them pass through the tunnel and, therefore, securing all the traffic at the same time.

And because I have a WR703N with an OpenVPN server installed, I decided to set up stunnel and give it a try. The advantage over using my existing VPN, under certain circumstances, is that the establishment of the secure tunnel looks pretty much like a normal connection to an HTTPS website so most of the networks/proxys will allow this traffic whilst the VPN might be blocked (especially if UDP is used). So, the OpenVPN+stunnel combo looks like a pretty good security solution to be installed on our OpenWRT device.

The way I have the stunnel service configured is using MTLS (client and server authentication) and allowing only TLSv1.2 protocol. These are the specific lines in the stunnel.conf (server side):

; protocol version (all, SSLv2, SSLv3, TLSv1)
sslVersion = all
options = NO_SSLv2
options = NO_SSLv3
options = NO_TLSv1

Just for testing, I have installed stunnel on a Windows box and configured it as a client (with a client certificate signed by the same CA as the server) and connections to server port 443 will be forwarded to the SSH service running on the server side. This would allow us to SSH our server without
needing to expose it and, for example, set up a SOCKS proxy and browse the internet securely through the tunnel.

stunnel Diagram

Client side:

accept  = 22
protocol = connect
connect = proxy:8080
protocolHost= server:443

Server side:

accept  = 443
connect = 22
TIMEOUTclose = 0

On the client side, simply SSH localhost on the configured port (22) and stunnel will intercept this connection and establish a TLS tunnel with the server to the SSH service running on it.

These are the logs on the client side when SSH'ing localhost:

2016.07.20 21:37:09 LOG7[12]: Service [https] started
2016.07.20 21:37:09 LOG5[12]: Service [https] accepted connection from
2016.07.20 21:37:09 LOG6[12]: s_connect: connecting proxy:8080
2016.07.20 21:37:09 LOG7[12]: s_connect: s_poll_wait proxy:8080: waiting 10 seconds
2016.07.20 21:37:09 LOG5[12]: s_connect: connected proxy:8080
2016.07.20 21:37:09 LOG5[12]: Service [https] connected remote server from x.x.x.x:43859
2016.07.20 21:37:09 LOG7[12]: Remote descriptor (FD=732) initialized
2016.07.20 21:37:09 LOG7[12]:  -> CONNECT server:443 HTTP/1.1
2016.07.20 21:37:09 LOG7[12]:  -> Host: server:443
2016.07.20 21:37:09 LOG7[12]:  -> Proxy-Authorization: basic **
2016.07.20 21:37:09 LOG7[12]:  -> 
2016.07.20 21:37:09 LOG7[12]:  <- HTTP/1.1 200 Connection established
2016.07.20 21:37:09 LOG6[12]: CONNECT request accepted
2016.07.20 21:37:09 LOG7[12]:  <- 
2016.07.20 21:37:09 LOG6[12]: SNI: sending servername: server
2016.07.20 21:37:09 LOG7[12]: SSL state (connect): before/connect initialization
2016.07.20 21:37:09 LOG7[12]: SSL state (connect): SSLv3 write client hello A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server hello A
2016.07.20 21:37:11 LOG7[12]: Verification started at depth=1: C=ES, ST=M, O=O, CN=wrtServer
2016.07.20 21:37:11 LOG7[12]: CERT: Pre-verification succeeded
2016.07.20 21:37:11 LOG6[12]: Certificate accepted at depth=1: C=ES, ST=M, O=O, CN=wrtServer
2016.07.20 21:37:11 LOG7[12]: Verification started at depth=0: C=ES, ST=S, O=O, CN=wrtClient
2016.07.20 21:37:11 LOG7[12]: CERT: Pre-verification succeeded
2016.07.20 21:37:11 LOG5[12]: Certificate accepted at depth=0: C=ES, ST=S, O=O, CN=wrtClient
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server certificate A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server key exchange A
2016.07.20 21:37:11 LOG6[12]: Client CA: C=ES, ST=M, O=O, CN=wrtCA
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server certificate request A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server done A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write client certificate A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write client key exchange A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write certificate verify A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write change cipher spec A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 write finished A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 flush data
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read server session ticket A
2016.07.20 21:37:11 LOG7[12]: SSL state (connect): SSLv3 read finished A
2016.07.20 21:37:11 LOG7[12]:      8 client connect(s) requested
2016.07.20 21:37:11 LOG7[12]:      7 client connect(s) succeeded
2016.07.20 21:37:11 LOG7[12]:      0 client renegotiation(s) requested
2016.07.20 21:37:11 LOG7[12]:      2 session reuse(s)
2016.07.20 21:37:11 LOG6[12]: SSL connected: new session negotiated
2016.07.20 21:37:11 LOG7[12]: Deallocating application specific data for addr index
2016.07.20 21:37:11 LOG6[12]: Negotiated TLSv1.2 ciphersuite ECDHE-RSA-AES256-GCM-SHA384 (256-bit encryption)

As you can see, the traffic will be routed through a TLSv1.2 channel encrypted with AES256 in GCM mode and the session key has been derived using ephimeral ECDH, with Perfect Forward Secrecy so the traffic will be fairly well protected, at least, up to the stunnel server.

Make sure to keep an eye on the vulnerabilities listed on the stunnel website and have the server properly patched.

Building RPM packages

I wanted to learn how to build an RPM package out of a Python module so, now that I'm playing a bit with OpenStack, I decided to pick up a log merger for OpenStack files and build the corresponding package on my Fedora 24.

First thing is to setup the distribution with the right packages:

[root@localhost ~]$ dnf install @development-tools fedora-packager
[dani@localhost ~]$ rpmdev-setuptree
[dani@localhost ~]$ ls rpmbuild/

Now, under the SPECS directory, we need to create the spec file which will include the necessary info to build the RPM:

%global srcname os-log-merger
%global	sum	OpenStack Log Merger

Name:		python-%{srcname}
Version:	1.0.6
Release:	1%{?dist}
Summary:	%{sum}

License:	Apache

BuildRoot:      %{_tmppath}/%{srcname}-%{version}-build
BuildArch:	noarch
BuildRequires:	python2

A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

%package -n %{srcname}
Summary:	%{sum}
%{?python_provide:%python_provide python2-%{srcname}}

%description -n %{srcname}
A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

%autosetup -n %{srcname}-%{version}


%{__python2} test

%files -n %{srcname}
#%license LICENSE
%doc README.rst

* Tue Jul 19 2016 dani - 1.0.6-1
- First version of the os-log-merger-package

Once the file is created, it's time to build the RPM package:

[dani@localhost SPECS]$ rpmbuild -bb os-log-merger.spec 
+ umask 022
+ cd /home/dani/rpmbuild/BUILD
+ cd os-log-merger-1.0.6
+ /usr/bin/rm -rf /home/dani/rpmbuild/BUILDROOT/python-os-log-merger-1.0.6-1.fc24.x86_64
+ exit 0
[dani@localhost SPECS]$ ls -alh ../RPMS/noarch/
total 44K
drwxr-xr-x. 2 dani dani 4,0K jul 19 20:35 .
drwxr-xr-x. 3 dani dani 4,0K jul 19 20:35 ..
-rw-rw-r--. 1 dani dani  34K jul 19 20:47 os-log-merger-1.0.6-1.fc24.noarch.rpm

We can see that the rpmbuild command produced the RPM file inside ~/rpmbuild/RPMS/noarch. Let's pull the info from it and check whether it's correct:

[dani@localhost SPECS]$ rpm -qip ../RPMS/noarch/os-log-merger-1.0.6-1.fc24.noarch.rpm 
Name        : os-log-merger
Version     : 1.0.6
Release     : 1.fc24
Architecture: noarch
Install Date: (not installed)
Group       : Unspecified
Size        : 85356
License     : Apache
Signature   : (none)
Source RPM  : python-os-log-merger-1.0.6-1.fc24.src.rpm
Build Date  : mar 19 jul 2016 20:47:42 CEST
Build Host  : localhost
Relocations : (not relocatable)
URL         :
Summary     : OpenStack Log Merger
Description :
A tool designed to take a bunch of openstack logs across different projects, and merge them in a single file, ordered by time entries

The last step is trying to install the actual file and execute the module to see if everything went fine:

[root@localhost noarch]$ rpm -qa | grep os-log-merger
[root@localhost noarch]$ rpm -i os-log-merger-1.0.6-1.fc24.noarch.rpm 
[root@localhost noarch]$ oslogmerger 
usage: oslogmerger [-h] [-v] [--log-base  LOG_BASE]
                   [--log-postfix  LOG_POSTFIX] [--alias-level ALIAS_LEVEL]
                   [--min-memory] [--msg-logs file[:ALIAS] [file[:ALIAS] ...]]
                   [--timestamp-logs file[:ALIAS] [file[:ALIAS] ...]]
                   log_file[:ALIAS] [log_file[:ALIAS] ...]


Simple 433MHz Keyfob

After the last two posts, I decided to build a simple PCB to handle various remotes in a single device and also serve as a "general purpose keyfob". I've built it around a PIC12F1840 microcontroller which I had handy. This microcontroller includes an internal oscillator so the external components were reduced to a minimum: just the radio transmitter, some push-buttons and a LED.


Initially, I aimed to power the board directly from a 1-cell Lipo battery (or 3 AA/AAA) but I included support for a higher voltage supply just in case the transmission power was too weak. If you want to power the keyfob with a higher voltage, just solder the regulator and adjust the resistors (R1, R3 and R5) so that the PIC reads no more than VCC volts at its inputs. For my application, it works just fine with a 4.2 battery and I get around 30 meters of transmission distance.

The board remains unpowered until the user presses a button. At that time, the microcontroller boots up, reads which of the button's been pressed and executes the action until released. Theoretically, a single battery should last a few years.

Below you can download the Kicad files so that you can modify anything, as well as the gerber files to order it your own. The transmission module can be easily found on many sites for less than $2.

Keyfob KICAD Files

Keyfob GERBER Files

Below is the C code for the microcontroller that can be compiled using the free version of MPLAB IDE. It's an example of how to send a different command depending on which button is pressed by the user.

 * File:   main.c
 * Author: Dani
 * Created on 7 de marzo de 2016, 20:06

#include <stdio.h>
#include <stdlib.h>

#include <xc.h>

#define _XTAL_FREQ 16000000 

// #pragma config statements should precede project file includes.
// Use project enums instead of #define for ON and OFF.

#pragma config FOSC = INTOSC    // Oscillator Selection (INTOSC oscillator: I/O function on CLKIN pin)
#pragma config WDTE = OFF       // Watchdog Timer Enable (WDT disabled)
#pragma config PWRTE = OFF      // Power-up Timer Enable (PWRT disabled)
#pragma config MCLRE = ON       // MCLR Pin Function Select (MCLR/VPP pin function is MCLR)
#pragma config CP = OFF         // Flash Program Memory Code Protection (Program memory code protection is disabled)
#pragma config CPD = OFF        // Data Memory Code Protection (Data memory code protection is disabled)
#pragma config BOREN = ON       // Brown-out Reset Enable (Brown-out Reset enabled)
#pragma config CLKOUTEN = OFF   // Clock Out Enable (CLKOUT function is disabled. I/O or oscillator function on the CLKOUT pin)
#pragma config IESO = ON        // Internal/External Switchover (Internal/External Switchover mode is enabled)
#pragma config FCMEN = ON       // Fail-Safe Clock Monitor Enable (Fail-Safe Clock Monitor is enabled)

#pragma config WRT = OFF        // Flash Memory Self-Write Protection (Write protection off)
#pragma config PLLEN = ON       // PLL Enable (4x PLL enabled)
#pragma config STVREN = ON      // Stack Overflow/Underflow Reset Enable (Stack Overflow or Underflow will cause a Reset)
#pragma config BORV = LO        // Brown-out Reset Voltage Selection (Brown-out Reset Voltage (Vbor), low trip point selected.)
#pragma config LVP = ON         // Low-Voltage Programming Enable (Low-voltage programming enabled)


 * RA0 = S1 (INPUT) LEFT
 * RA1 = S2 (INPUT) INH
 * RA5 = Tx (OUTPUT)
 * RA4 = Testpoint

#define TX_PIN  RA5

#define PKT_LENGTH  48

const unsigned char left_pkt[]   = {0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA};
const unsigned char right_pkt[]  = {0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,0x00,0x00,0x00,0xAA,0xAA,0xAA,0xAA,0x00,000,0x00,0x00};

#define BITDELAY    70

#define delayMicroseconds   __delay_us
void tx_frame(const unsigned char *ptr, unsigned int length)
    unsigned int i;
        TX_PIN = ((ptr[i] & 0x80) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x40) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x20) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x10) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x08) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x04) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x02) != 0);    delayMicroseconds(BITDELAY);
        TX_PIN = ((ptr[i] & 0x01) != 0);    delayMicroseconds(BITDELAY);

int main(int argc, char** argv)
  unsigned char left=0, right=0, extra=0;
  OSCCON |= (0x0F<<3);  // Internal oscillator @ 16MHz
  ANSELA = 0;                       // No analog inputs 
  WPUA=0;                           // No internal pullups
  TRISA |= ((1<<0)|(1<<1)|(1<<2));  // Buttons as Inputs
  TRISA &= ~(1<<4);                 // RA4 (testpoint) as Output PIN

  // Allow some time to set things up
  //Check what actions are requested:
  if((PORTA & 1) == 1)  left    = 1;
  if((PORTA & 2) == 2)  extra   = 1;
  if((PORTA & 4) == 4)  right   = 1;
  return (EXIT_SUCCESS);

Below, a couple of pictures of the board with the transmission module attached:









I'll build a second version of the board to fit a 3-button case or maybe I get one case 3D printed 🙂


RFCat, TI Chronos and replaying RF signals :)

After my first contact with the RTL-SDR a couple of days ago , I've been researching a bit more and found this fantastic blog post by Adam Laurie which describes how to use a TI Chronos development kit to send arbitrary sub-1GHz signals. It happens that I had such a kit so I decided to emulate another garage door opener, but this time using RFCat.

Loading RFCat firmware into the Chronos USB Dongle

First thing I did was to flash the USB dongle with the RFCat firmware so that I could emulate the remote from a python script. As I had a CC Programmer handy (you can also use GoodFET), I wired it up by following the diagram below and flashed the RFCat bin for the ez Chronos dongle using the SmartRF Flash Programmer tool.






You can either flash the dongle with the RFCat binary itself or with the CC Bootloader which will allow you to update the dongle further without having to use the JTAG. I took the second approach so after flashing the bootloader, you'll need to flash the actual RFCat firmware:

python /dev/ttyACM0 download RfCatChronosCCBootloader-150225.hex

After successfully flashing the dongle, it should show up as "RFCat" and you should be able to communicate with it from the rfcat interpreter:


As the communication with the dongle was good, it was time to analyze the signal sent by the remote and write some code to replay it using RFCat.

Signal Analysis

For the analysis part, I used the SDR# tool for Windows: tuned the right frequency (433.92MHz) and saved the signal into a Wav file for later analysis with Audacity.


It's a fixed code and looks pretty straightforward: short and long pulses. We can estimate the length of each type by measuring the number of samples. In this case, short pulses took 3000 samples or 1200us (sample rate was 2.4Ms on SDRSharp).

A good way to represent the signal is to encode the "long pulse plus silence" as "1" and the "short pulse plus silence" as "0". Then, the frame would look like this:

1  0  1  1  0  1  1  1  1  1  1  0  0  0  0  0  1  0  1  1  0  0  1  1  0  0  1  1  0  0  1  1  1

As the "1" is formed by two high and one low short pulses of equal duration, we can express it as "110". Similarly, our "0" can be represented as "100" and the frame now would be:

110 100 110 110 100 110 110 110 110 110 110 100 100 100 100 100
110 100 110 110 100 100 110 110 100 100 110 110 100 100 110 110 110

However, if we zoom in on the signal, we can see that the pulses are divided in more little pulses that we'll need to encode in some way:


So, the final frame would make us rewrite the previous one changing every "1" bit by \xAA\xAA  and every "0" bit by \x00\x00 to maintain the length of each bit (see code below).  The duration of each bit is now about 80 us.

Replaying signal with RFCat

Now that we have analyzed the signal, it's time to write a Python script to interface the RFCat dongle so that it generates the frames accordingly. Afterwards, we'll capture the signal back to make sure that both the waveform and timing are correct:

from rflib import*
from time import sleep

pkt = '\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\x00\x00\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00\xAA\xAA\xAA\xAA\x00\x00'

NUM_REPS	= 10		# times the frame will be sent
DELAY 		= 0.02	# seconds between frames


	d = RfCat()
	d.setFreq(433290000)	# Set freq to 433.92MHz
	d.setMdmSyncMode(0)		# Don't send preamble/sync word
	d.setMdmDRate((int)(1.0/0.000080))	# Our bits are 80us long

	print "Sending frames "
	for i in range(0,NUM_REPS):
	print " Done\n"

except Exception, e:
	sys.exit("Error %s" % str(e))

Now let's run the script and capture the signal back with SDR# to check if it looks like it should:



The first picture shows both our reference signal (sent by the remote) and the one generated with the RFCat dongle. The second picture shows the detail of each bit. As expected, it opened the door although the output power seemed a bit too low . Maybe there's a hack to improve the antenna of the Chronos dongle?  🙂

Replaying the signal with the Chronos Sports Watch

Okay, the hard part is over so let's have some fun and replay the signal directly from our wrist 🙂

The ChronIC project is pretty much like the RFCat firmware but can loaded directly into the Chronos Sports Watch so that pre-loaded signals can be sent just by pressing the up/down buttons. I modified the code to make the watch send our frame every time I pressed the UP button. Below is the code that will do the magic, and a couple of useful Python functions (from Adam's code) to calculate the register values for your bitrate and frequency:


 def setfreq(freq):
	mhz= 26
	freqmult = (0x10000 / 1000000.0) / mhz
	num = int(freq * freqmult)
	freq0= num & 0xff
	payload= chr(freq0)
	freq1= (num >> 8) & 0xff
	payload += chr(freq1)
	freq2= (num >> 16) & 0xff
	payload += chr(freq2)
	print '- FREQ2: %02x FREQ1: %02x FREQ0: %02x -' % (freq2, freq1, freq0)

def setdatarate(drate):
	mhz= 26
	drate_e = None
	drate_m = None
	for e in range(16):
		m = int((drate * pow(2,28) / (pow(2,e)* (mhz*1000000.0))-256) + .5)        # rounded evenly
		if m < 256:
			drate_e = e
			drate_m = m
	if drate_e is None:
		return False, None
	drate = 1000000.0 * mhz * (256+drate_m) * pow(2,drate_e) / pow(2,28)
	print 'drate_e: %02x  drate_m: %02x' %(drate_e,drate_m)
void config_garage(u8 line)
	// gap between data pulses
	//Button_Delay= 0;
	Button_Delay= 20;
	// how many times to send per button press
	Button_Repeat= 10;

	// set button content

	Up_Buttons= 1;
	// packet length
	Button_Up_Data[0][0]= 198;
	// payload

	Down_Buttons= 0;

	// set frequency (433920000)
	ChronicRF.freq0= 0x71;
	ChronicRF.freq1= 0xB0;
	ChronicRF.freq2= 0x10;

	// set data rate (pulsewidth 80us)
	// drate_m
	ChronicRF.mdmcfg3= 0xf8;
	// drate_e
	ChronicRF.mdmcfg4 &= 0xf0;
	ChronicRF.mdmcfg4 |= 8;

	// set modulation
	ChronicRF.mdmcfg2 &= ~MASK_MOD_FORMAT;
	ChronicRF.mdmcfg2 |= MOD_OOK;
	// set sync mode
	ChronicRF.mdmcfg2 &= ~MASK_SYNC_MODE;
	ChronicRF.mdmcfg2 |= SYNC_MODE_NONE;
	// set manchester false
	ChronicRF.mdmcfg2 &= ~MASK_MANCHESTER;
	display_symbol(LCD_ICON_RECORD, SEG_ON);

After  building the code with Code Composer and loading it into the Watch with the JTAG included in the kit, a new menu is available and the signal's going to be sent every time we press the UP button.


🙂 🙂

All the information in this blog is for educational  purposes only.  You shall not misuse the information to gain unauthorized access.


A new cool gadget has fallen into my hands: a SDR DVB-T dongle based on the Realtek RTL2832U chipset. Little did I know I was gonna get so fascinated about this world and first thing I wanted to try is to open my garage door from an Arduino as a "Hello World" exercise.


Firstly, I installed the necessary software on a Kali Linux distribution and checked the frequency of the remote with the gqrx tool:


Then, I dumped the signal for further analysis into a wav file using the gnuradio-companion software:


This flowgraph will let us sample the signal sent by the remote and write it into a Wav file. After pressing the buttons on the remote we can see how it looks like in Audacity:

The modulation used by the remote is OOK and looks like it uses Manchester codification. Using the rtl_433 tool, I was able to decode the frame and correlate it with the waveform above:

*** signal_start = 2189802, signal_end = 2300238
signal_len = 110436,  pulses = 86
Iteration 1. t: 404    min: 110 (47)    max: 699 (39)    delta 241
Iteration 2. t: 404    min: 110 (47)    max: 699 (39)    delta 0
Pulse coding: Short pulse length 110 - Long pulse length 699

Short distance: 83, long distance: 675, packet distance: 4804

p_limit: 404
bitbuffer:: Number of rows: 6 
[00] {4} f0 : 1111
[01] {18} 23 23 c0 : 00100011 00100011 11
[02] {18} 23 23 c0 : 00100011 00100011 11
[03] {18} 23 23 c0 : 00100011 00100011 11
[04] {18} 23 23 c0 : 00100011 00100011 11
[05] {10} 23 00 : 00100011 00

As long as the button's pressed, the remote will keep on transmitting the 18-bit frame which we have identified as:   00100011 00100011 11. This bitstream can be clearly seen on Audacity.

From the output above, the short pulses got 110 counts and the long pulses 699. Since the rtl_433 tool samples at 250KHz, it means that they last 440us while the long ones last 2800us. All we need to do now is write the software for the Arduino board to replicate the signal:

#define rfTransmitPin 4
#define ledPin 13
#define buttonPin 9      

void setup(){

	pinMode(rfTransmitPin, OUTPUT);
	pinMode(ledPin, OUTPUT);
	pinMode(botonPin, INPUT);       

	digitalWrite(rfTransmitPin, LOW);

  void loop(){
    if(digitalRead(buttonPin) == HIGH)   // if the button is pressed, tx the code

#define SHORT_WAIT	delayMicroseconds(440)
#define LONG_WAIT	delayMicroseconds(2800)
#define TX_LOW		digitalWrite(rfTransmitPin, LOW)
#define TX_HIGH		digitalWrite(rfTransmitPin, HIGH)

#define FRAME_SIZE        18

unsigned char code[] = {0,0,1,0,0,0,1,1,0,0,1,0,0,0,1,1,1,1};

void transmitCode() {

    digitalWrite(ledPin, HIGH);

	for(int i=0;i<CODE_SIZE;i++)
		if(code_left[i] == 1)
	digitalWrite(rfTransmitPin, LOW);

    digitalWrite(ledPin, LOW);

Now, let's download it into the microcontroller and capture what it's sent to see if it matches the original code:

Arduino Sample



The waveform looks pretty much the same as the one sent by the remote. Also, the rtl_433 tool is able to decode it properly and the timing looks quite nice too:

*** signal_start = 12434285, signal_end = 12748315
signal_len = 314030,  pulses = 144
Iteration 1. t: 413    min: 115 (84)    max: 711 (60)    delta 8
Iteration 2. t: 413    min: 115 (84)    max: 711 (60)    delta 0
Pulse coding: Short pulse length 115 - Long pulse length 711
Short distance: 112, long distance: 708, packet distance: 25527

p_limit: 413
bitbuffer:: Number of rows: 8
[00] {18} 23 23 c0 : 00100011 00100011 11
[01] {18} 23 23 c0 : 00100011 00100011 11
[02] {18} 23 23 c0 : 00100011 00100011 11
[03] {18} 23 23 c0 : 00100011 00100011 11
[04] {18} 23 23 c0 : 00100011 00100011 11
[05] {18} 23 23 c0 : 00100011 00100011 11
[06] {18} 23 23 c0 : 00100011 00100011 11
[07] {18} 23 23 c0 : 00100011 00100011 11

Now we're sure that the Arduino board will transmit the same signal, it's time to try it by the garage door and... it WORKS! 🙂

It is a very simple project but as a first contact with the RTL-SDR world I had a lot of fun. I'm looking forward to learning more about it, especially the gnuradio-companion software for signal processing and analysis.


OpenVPN & OpenWRT - Secure Browsing from your mobile phone

Some time ago, I bought a TL-WR703N WiFi router for less than 15€. It came with a Chinese firmware that I overwrote with an OpenWRT image and connected to my ADSL router.

This device is really cool but once you flash it with an OpenWRT image you'll find out that there's almost no free space  (4MB total flash) so I decided to use an external USB memory to increase the available space and turn it into a useful gadget 🙂


I used to have stunnel and OpenVPN servers running on my PC but since I didn't want to have the computer on all day, I decided to replace it with this small device which makes no noise and consumes very low power (around 80mA/~0.4W with WiFi on if I disable the blue LED :))

First thing I did was setting up an AP and disable my router's WiFi network since its antenna was surprisingly better and my old router doesn't support WiFi n. So my devices at home would connect to the WR703N WiFi network which was bridged with the ethernet interface to the ADSL router.

After this introduction, I'll explain how to set up an OpenVPN server to browse securely anywhere from your phone which is especially useful if you're using free or untrusted wifi networks out there.


OpenVPN Diagram

OpenVPN Diagram

At the moment of writing, I'm on the latest OpenWRT version which is Attitude Adjustment 12.09, r36088.

Required packages:

  • openvpn-easy-rsa - 2.2.2-2 - Simple shell scripts to manage a Certificate Authority
  • openvpn - 2.2.2-2 - Open source VPN solution using SSL


1. Certificates and keys generation

The easy-rsa package helps you create the CA, server and client certificates but you can either create them yourself or use existing ones created somehow (as long as you keep your private keys secret 🙂

I created 2048 bit RSA certificates with the help of the easy-rsa tool:

Edit the /etc/easy-rsa/vars file and change the KEY_SIZE value to 2048

export KEY_SIZE=2048

Also, feel free to change the certificate public data such as the common name, country, etc.


root@OpenWrt:/etc/easy-rsa# build-ca
NOTE: If you run ./clean-all, I will be doing a rm -rf on /etc/easy-rsa/keys
Generating a 2048 bit RSA private key
writing new private key to 'ca.key'

Now generate the DH parameters and the server and client certificates signed by the previous CA.

build-key-server server
build-key-pkcs12 daniiphone

Afterwards, all those files will be located under /etc/easy-rsa/keys and must be copied over to the openvpn directory:
root@OpenWrt:/etc/easy-rsa/keys# cp ca.crt ca.key dh2048.pem server.crt server.key /etc/openvpn/

The client certificate is not needed on the server side but we've generated it inside the WR703N and must be copied onto the client (in this case, my iPhone). We've generated the client certificate (daniiphone) in PKCS12 format which includes both the public certificate and the private key. It's really important to protect it with a password because you're gonna send it to your e-mail in order to import it from the OpenVPN iOS app. At the time of creating it, you'll be prompted to enter a password.


2.  Server side configuration

I've configured the OpenVPN server as follows using uci (you can do it by editing /etc/config/openvpn file).

root@OpenWrt:/etc/openvpn# uci show openvpn
openvpn.myvpn.keepalive=10 120
openvpn.myvpn.push=route dhcp-option DNS dhcp-option DOMAIN

These settings tell the server to listen on UDP port 1194 (which needs to be forwarded in your ADSL router to the WR703N IP address) and sets the VPN network at (clients will be assigned an IP address in this subnet).
The last line creates a default route to my lan network and shall be replaced with your own configuration.

Now we need to create a rule in the firewall to permit the VPN traffic. Add the following rule to the /etc/config/firewall file on the OpenWRT system:
config 'rule'
option 'target' 'ACCEPT'
option 'name' 'vpn'
option 'src' 'wan'
option 'proto' 'udp'
option 'dest_port' '1194'

In order to forward traffic from the VPN to the wan connection, we need to enable forwarding on the tun interface and create an NAT to the local interface:

iptables -I INPUT -i tun+ -j ACCEPT
iptables -I FORWARD -i tun+ -j ACCEPT
iptables -I OUTPUT -o tun+ -j ACCEPT
iptables -I FORWARD -o tun+ -j ACCEPT
iptables -t nat -A POSTROUTING -s -j SNAT --to-source

Replace the with your OpenWRT lan IP address and start the openvpn service:

Enable OpenVPN service autostart
root@OpenWrt:~# /etc/init.d/openvpn enable
Start the service
root@OpenWrt:~# /etc/init.d/openvpn start


2.  Client side configuration

Now we need to setup the iPhone (it should work on Android phones or any computer running OpenVPN) to connect to our VPN server. First we need to transfer the .p12 file created in the first step and import it to the phone from an e-mail attachment.

Install Certificate

Install Certificate

Certificate Installed

Certificate Installed

Once you have successfully imported the certificate file to the iPhone, it's time to load the OpenVPN configuration. In order to load it easily, create a .OVPN file like this one:

dev tun
proto udp
remote your-public-ip-address 1194
user nobody
group nogroup
resolv-restry infinite

Make sure you specify your public IP address and your CA certificate inside the configuration. This CA is needed because we have used a self-signed CA which is not trusted by the OS so if we didn't include this certificate within the configuration, the OpenVPN client would not trust the certificate presented by the server during the TLS negotiation.

This .OVPN file has to be imported from an e-mail attachment directly into the OpenVNP app. Once imported, click on the green "Add" button to associate the previous certificate and its private key to this profile. In your configuration you should be able to see your remote IP address (or hostname) instead of "80.80." which I've edited in the screenshots below.

Import OVPN file

Import OVPN file

Add imported certificate

Add imported certificate


OpenVPN Profile

OpenVPN Profile

OpenVPN connected

OpenVPN connected

Now that the tunnel has been setup, you should be able to see the "VPN" symbol on the status bar of your iPhone and all your traffic will be encrypted up to your home network. In order to test the connectivity and the forwarding rules, I try to access the OpenWRT Luci web configuration by typing the WR703N IP address in Safari:

OpenWRT Luci

OpenWRT Luci

Now, your iPhone is connected to your home network and all the traffic will go through your ADSL connection. Anyone trying to eavesdrop on the WiFi network will only be able to see tons of encrypted traffic.

It's very important that you keep the CA private key secret in order to avoid "man-in-the-middle" attacks, as well as protect the .p12 file when you send it over to your phone.

L8 SmartLight - Our Kickstarter adventure!

It’s been quite a long time since I don’t update this blog. The reason why is that I’ve been too busy with L8 Smartlight, a new entrepreneurship project that some friends and I have started a few months ago.

We’re very proud of being the first Spanish project to be successful on Kickstarter and, now that the funding project is about to finish, we’re very keen on starting the production of this new gadget and looking forward to spreading the world with it!

So far, the project’s had sparked the interest of many important sites such as Mashable:

We’re currently working on fully supporting Bluetooth EDR and 4.0 to cover most of the smartphones in the Market. Also there’s still a long way to go to define all the APIs and functionality that will be hopefully ready in the next few months.

I’ll try to update the status on the L8 project in this blog but, we’ll make sure that the latest information will be available right away at Kickstarter


Silvestre - Cosmobot 2012 Champion!

Silvestre Champion

Silvestre became champion of the Cosmobot 2012, one of the main robotics contests in Spain, after a really tough competition.

During the qualifying session in the morning, Silvestre made the fastest time with an average speed of 2.45 meters per second. Later in the evening, 16 robots went into the knockout phase and Silvestre had to make the most out of itself to push into an speed of 3 meters per second during the final round.

Knockout phase

The strongest point of Silvestre was, undoubtedly , his ability to learn the track and accelerate/brake at the right moments. Unlike past years, Silvestre doesn’t need any parameter to be entered manually but he will dynamically extract the key parameters of the track thanks to the encoders and inertial sensors. After the track’s been successfuly learnt, he will store it into memory for later runs in order to save the learning process every time. However if the data reported from the sensors is different from the expected track, he will learn again or modify it accordingly allowing him to self-adapt to changing conditions.

Cosmobot 2012 Track

The track was really good for our algorithm even though the zig-zag zone was tricky but also a good sucession of events that provided a more accurate positioning 🙂  The rules of the contest state that if a robot touches the internal lane, it would get disqualified but nothing prevents a smart robot from going through the zig-zag area as if it was a straight.

Silvestre was able to recognize the zig-zag area and, depending on the selected profile, he would “patch” the stored track to navigate inertially without reading the line, strictly during the zig-zag turns. This involves a risk because the top speed was about 6m/s and the navigation had to be as accurate as possible but it was robust enough so I used it on the final round :)



I don’t know what happened during the first round but Silvestre missed the line (Murphy’s law pushed him?) and started to learn the track again but, before this happened, the rival caught him up. The next two rounds, everything worked great and Silvestre won the championship!

After the contest, people wondered if Silvestre could learn the track clockwise and here’s the video:

Another video of Silvestre from another angle:

Slow motion video of the zigzag turns:

And below is the proof that what the videos show is not easy at all and took many hours analyzing data and trying:

Silvestre Wheels

New vs. used wheels after some training days

Unfortunatelly, my friend Alberto and I, don’t have as much time as we’d like to spend on the robots but we’re still enjoying competing and getting home at 2AM after a 20-hour of hard work the day before the competition 🙂  It’s great to see how racing robots have improved during the past two years and how innovative one must be in order to keep up pace.

UPDATE: video of the two semifinals and the final. Awesome!