Enough technology to get the job done

I live on a hill. If I knew that I was going to get into cycling maybe I would have found somewhere else but it’s mostly hills around here anyway. After a couple of years of commuting by force of will (most of the time) I finally caved and bought an e-bike.

The electric bike is a technological unicorn. Usually we make better technology by using more resources—fancier raw materials, better manufacturing, more software, more home delivery, more energy consumption. A lump of metal and plastic in the kitchen can wash the dishes. A Google datacentre full of computers can process my voice recording and provide some information in return.

Most of the time the newer technology is also more complex and fragile. Perhaps it requires recharging every day or an internet connection, and when something breaks the only source of spare parts is overseas.

Without a doubt an e-bike is more complex and fragile than a normal pushbike. It has a lithium-ion battery with a limited life, an electric motor and a little computer that applies rotational power in proportion to my pedalling effort. I’d better be getting something good in return.

Here’s the payoff: now I can get to my home without excessive physical effort, and also without bringing hundreds of kilograms of extra weight propelled by fossil fuels. Until a few years ago this just wasn’t possible. This is something that I need to do almost every day of my life. It’s a huge improvement to my basic living efficiency and it’s now reasonably affordable.

Of course, self-driving cars are the future. Well, that’s what the media tells me anyway. They’re probably right. Human drivers are pretty bad, and there are lots of families and situations where bicycles are not an appropriate way to get around. But they’re complex: very complex computers and sensors using GPS satellites and the Internet. It will be effective but it won’t be elegant.

Nothing is wasted when I ride up my hill. The motor can’t quite propel me by itself. It gives its all and I provide the rest, which isn’t much. It charges with less energy than it takes me to cook dinner. It’s 21 kilograms of bike and a battery that I’ll have to replace in a few years.

It’s just enough technology to get the job done, and no more. This makes me happy.

Posted in tasmania | Leave a comment

Synapse matrix homeserver with systemd

I run a Matrix homeserver and until recently I was using the virtualenv + synctl method to start the service, which they describe in the README. What I really wanted was to put it into systemd so that I can start and stop it like any other system service.

There is a basic synapse.service file in the synapse repository for Arch Linux. It needs tweaking a bit to work with a virtualenv installation. In my case I have a user called synapse and all the homeserver files are in /home/synapse.

I created a file /etc/systemd/system/synapse.service:

The main trick is to use the python associated with the virtual environment so that it can find the homeserver and all its dependencies. By running python directly systemd is able to track that the process is running – if you use synctl start it forks off into the background shortly after and systemd thinks it failed.

Then it’s just matter of:


Posted in sysadmin | Leave a comment

An ARP Server in Rust – Linux Raw Sockets

Linux has a fiddly but useful C interface for sending and receiving raw network packets. It’s documented primarily in the packet(7) manpage and the workhorse of this interface is a struct called sockaddr_ll. It’s ostensibly a type of sockaddr but it has a different length, different fields, and different parts that are relevant in different situations.

This is not especially ergonomic in Rust. The libc bindings for recvfrom(), for example, require you to pass in a *mut sockaddr. If you’re receiving from a raw socket then this actually needs to be a pointer to a sockaddr_ll, which is included in libc but is a completely different type from Rust’s point of view. Clearly Rust’s type checker is not going to be thrilled about this.

I recently figured out how to make this work for sending and receiving raw AX.25 packets. I wanted to blog about how I got it working, but since AX.25 is a little obscure I’ve decided to build a basic ARP server using similar code. Hopefully it makes a good example. This is the complete code for the server I’m going to be describing: https://gist.github.com/thombles/16736c9c656e6dad9a08c81b30a974ac

Opening A Socket

To send and receive packets the first thing you need is a socket. If you’re using the AF_PACKET interface you can choose between SOCK_RAW and SOCK_DGRAM. The first includes the ethernet headers in the packet body; the second does not. I’m using SOCK_RAW so when I receive an ARP packet so this is the information I will end up receiving:

To open the socket I use the socket() function from libc and some of the standard constants. You can choose to filter by a particular protocol by using one of the constants from the Linux header file if_ether.h. In this case I’m using ETH_P_ARP because I only want to know about ARP packets

The ETH_P_ARP parameter must be handled carefully. Ethernet protocol identifiers are 2 bytes, so it’s a u16. However Linux requires that we provide this parameter in network byte order, which is big-endian. The parameter to socket() is then defined as a c_int, which means an i32. It’s important to do these steps in the right order. This is how it works on my Intel processor. If you happened to be using a big-endian platform it would be 0806 all the way through.

Overall flow

This is an unintelligent ARP server and it will work like this:

  • Set up a HashMap of IPv4 addresses that we will respond to, with each entry storing the corresponding MAC address.
  • Listen to all incoming ARP packets on all interfaces. If it is an ARP query for an address in our HashMap, send a reply, acting as that computer.

This won’t be useful for much more than ARP spoofing since anything that knows how to talk IP will generally take care of its own ARP. But it will do for a demo. In this code I’m just using a hard-coded HashMap that’s configured when the app launches.

The following will open the socket then attempt to receive a packet. Each time it does, if it was not an error it will hand it off to another function to read.

For each received packet we get back two blobs of data. One is the actual packet. The other is a chunk of metadata, the sockaddr_ll. In this case I am (slightly arbitrarily) using two different strategies for receiving these chunks of data.

The sockaddr_ll is declared on the stack and I’m passing in a mutable pointer, so the C function can fiddle with the contents of the on-stack memory. Later on, after the unsafe code, hopefully the zeroed fields have been replaced with something valid.

The packet content is defined as a simple buffer of u8. I could probably use my RawArpFrame struct the same as sockaddr_ll, but keeping it as an untyped buffer means I could potentially handle different kinds of protocols. I will instead turn the buffer into a RawArpFrame inside handle_packet().

Receiving an individual frame

The recvfrom() function will block until it it either successfully receives a frame or suffers an error. This function wraps recvfrom() and returns the number of valid bytes that have been written into the buffer.

This is where it gets a little nasty. I can’t create a *mut sockaddr from a sockaddr_ll; they are simply different types. I use the unsafe transmute function to convert the pointer from one type to another. That transmuted pointer can then be used as a parameter to recvfrom(). On the other hand, the data buffer’s mut ptr can simply be cast to an opaque *mut c_void as required for the libc parameter.

Parsing the incoming frame

Once an ARP frame has been received we have two pieces of information: the sockaddr_ll “metadata” provided by Linux, and an opaque blob of binary data in a buffer. The handle_packet() function converts that opaque blob into a struct with fields and decides if we want to reply to it or not.

Our sockaddr_ll was written into directly during the recvfrom() so we can simply use it. The buffer takes more work. Here, ptr::read takes a pointer and uses it to construct a real RawArpFrame based on that chunk of memory. We have to hope that the layout in memory matches the definition of the struct, which is why it’s unsafe.

Once we have that frame it’s straightforward to decide whether we should reply to it or not.

Sending an ARP response

Sending a raw packet in Linux requires filling in particular fields of the sockaddr_ll as documented in the manpage. The most tricky of these is the interface index, which identifies which network interface on the machine should be used to send the data. You access this with the SIOCGIFINDEX ioctl, as described in netdevice(7). To find out which interfaces exist you can either read /proc/net/devices (more complete if you don’t always have IP addresses) or use the SIOCGIFCONF ioctl. (Example code for this ifindex discovery.)

It is possible to sidestep a lot of this work when you only wish to reply to incoming packets on the same interface that you received them. Happily, this is the case for ARP. All the information you need is contained in the sockaddr_ll from the packet you received.

The top of the function which manipulates frame is simply shuffling fields around to turn it into a valid ARP response, including the missing MAC address.

It then creates a new sockaddr_ll which is mostly initialised to zero but takes the key fields from the received sockaddr_ll. It likely wouldn’t hurt to include more of the fields. The documentation is not crystal clear so I’m not taking my chances.

The call to sendto() is very similar to what we saw before with recvfrom(). We have to pass both a sockaddr and the data to be transmitted. The difference this time is that both of these sources of data are strongly typed structs.


The snippets above cover the main functionality. I want to mention a few other things I found along the way.

When you use the netdevice ioctls you are forced to work with an ifreq union. This is an especially tricky data structure that can easily get messy. In my own code I used a technique by Herman J. Radtke III that makes this reasonably easy to handle without using unsafe.

Normally AF_PACKET/SOCK_RAW will only give you incoming packets. Word on Stack Overflow is that using ETH_P_ALL instead of a specific protocol will include outgoing packets, but then it will be necessary to filter by protocol yourself.

Finally, capturing packets promiscuously and sending raw packets requires either root access or the capabilities CAP_NET_ADMIN and CAP_NET_RAW. If you see permission denied errors, this is probably why.

Posted in programming, rust | Leave a comment

Improving ARP performance on slow AX.25 links

Recently I’ve been setting up TCP/IP connections over VHF radio. They are Linux systems linked by 1200 baud AX.25 modems. Opinions vary as to whether the overheads of TCP/IP are worthwhile at 1200 baud—9600 is certainly better—but it’s handy and it definitely works provided you use the available bandwidth carefully. All IP communications need to be precise and tidy. Otherwise the channel gets gummed up for seconds at a time and it’s not much fun to use.

Using Debian/Raspbian one of the most obvious problems is ARP traffic. When I watch the channel with axlisten I see huge quantities of requests and replies flying past, maintaining associations between callsigns like “VK7NTK-1” and IP addresses like*.

At home I have two separate computers and radios set up to talk to each other. With nothing in the ARP cache, if I run ping -c 1 I would naively expect to see something like the following. It’s basically like Ethernet except instead of MAC addresses there are callsigns and SSID numbers.

First the pinging station finds out who owns the IP; then it sends the ping. Aah, if only.

Problem 1: Duplicate ARP requests

Using a default ARP configuration the transmissions actually turn out like this:

At 1200 baud it takes most of a second to transmit even the shorter packets. The modems also check that the channel is clear before transmitting, introducing additional delays. As a result, after VK7NTK-2 decided it needed to do an ARP query the other computer didn’t even finish receiving it until 0.8 seconds later. VK7NTK-1 transmitted its reply, taking another 0.7 seconds, so VK7NTK-2 had the address it needed about 1.5 seconds after its initial request. Unfortunately at the 1 second mark it decided that the first request had been lost and queued a second query for transmission.

When that first reply did come back at 1.5 seconds VK7NTK-2 realised it had the information it needed to do the ping so it created the ICMP packet and queued that for delivery too. At that point it had two packets to transmit: the second ARP query, now unnecessary, and the ping request. In this case the modem decided to transmit them back-to-back, which is allowed.

VK7NTK-1 received both of these and queued up a reply to each one. These were also transmitted back-to-back and the ping ultimately succeeded.

The basic problem is that 1 second is far too short a timeout for ARP requests at 1200 baud. Linux isn’t clever enough to cancel the second ARP query when it’s no longer needed so I end up with an entire extra request and reply.

But wait, there’s more! Several seconds after the ping there is suddenly an ARP query in the other direction, originating from the computer that received the ping. I’m guessing this is an optimisation in Linux to warm up the ARP cache in advance. This one was even messier:

In total VK7NTK-1 transmitted three ARP requests and VK7NTK-2 transmitted two ARP replies. Two of those transmissions doubled—that is, both modems thought the channel was clear and started transmitting simultaneously. VK7NTK-1 didn’t receive the answer it wanted because it was too busy shouting the question over the top.

This is also caused by the 1 second timeout but in a different way. If you send out a query that takes 3/4 of a second to transmit, then add a little time for processing and making sure the channel is clear, you would assume that the reply will begin at about the 1 second mark. For the sender of the query, 1 second later is about the worst possible time to make a second transmission, regardless of whether it has anything to do with the previous one. You’re just asking for a double.

Solution 1. Adjust the the ARP retransmission time

The good news is that Linux makes it easy to change the ARP retransmission time. The following command increases it from the default 1 second to 5 seconds:

I placed this in a script that runs after the ax0 interface is configured. It solves both of the above problems fine.

Problem 2: The forgetful cache

After all the nonsense they went through exchanging ARP packets it would be nice if Linux remembered the results for a while. Unfortunately it doesn’t. It’s slightly randomised but roughly 15 seconds later the ARP entry will change from “REACHABLE” to “STALE”. A subsequent ping will cause the whole schmozzle to happen again. It wouldn’t be a big deal on zippy Ethernet but it really is on 1200 baud.

You can check the state of ARP entries using this command:

Solution 2: Increase the reachable time

This is another parameter that’s easy to control. To have 10+ minutes in the REACHABLE state:

The best solution: static ARP entries

Unlike Ethernet where most people are using DHCP and changing addresses semi-regularly, AX.25 stations tend to have pretty fixed addresses. Adding a permanent ARP entry means that Linux never makes any requests at all for that station.

I use this command to add an entry for my upstream gateway VK7HDM-6:

It is still good to have the other optimisations in place for stations that come and go.

* The problem looks slightly worse than it is because axlisten (or arguably the kernel) is a little buggy. The obsolete AF_INET raw interface seems to report duplicates of some frames. I have confirmed with my own software that the newer AF_PACKET/SOCK_RAW interface works fine. See manpage packet(7).

P.S. The very nature of doubled transmissions is that the computers don’t really know what’s going on and Wireshark gives misleading results. An iPad clock and phone video are a low-tech but dependable way to observe sub-second collisions.

Posted in radio | Leave a comment