Sending JSON over websockets in Rust

I like the JSON format but I feel it’s important to contain it to the job of passing a message from one system to another. Ideally the sender generates the text automatically from some strongly-typed object and the receiver does the reverse. If any required parts are missing or a field is the wrong type the entire message is rejected. Skipping this step and operating directly on the fields, as you can do in Javascript or with NSJSONSerialization, relies on the programmer not stuffing it up. I usually stuff it up.

I’m working on software that needs to send regular status updates over a TCP connection to a central server. JSON is a nice human-readable choice for the data format. Websockets are great because I can lean on an Apache proxy to provide LetsEncrypt-signed TLS, and unlike a raw TCP stream the communication is already packetised into discrete messages.

I felt like using Rust but had no idea how hard it would be compared with a dynamically typed language, so I prototyped it. It turned out relatively straightforward thanks to a couple of excellent crates, serde_json and ws-rs:

This is the data model, which is similar to what I have planned for the real app. To exercise the parser I made sure it has a few tricky bits: an enum with associated values, an array, and a struct of one type within another. I didn’t need to tweak them at all. The default implementations of Serialize and Deserialize cope perfectly.

The None type of Option is represented as a JSON null, and the serde parser will quietly provide a None if the field is missing entirely in the JSON. While serde_json supports various enum formats, without any customisation I already get a very nice representation for the overall PlayerState. (serde_json’s output is minified.)

Very little is required on the server. The listen function provided by ws-rs blocks, using an mio event loop under the hood to handle an arbitrary number of connected clients. Right now I’m only providing a handler for each incoming message. If I can parse the JSON I do so and print the debug representation of the PlayerState.

As I hoped, the PlayerState is fully filled in from the JSON:

Over in the client it needs to juggle both the websocket connection and a timer to send the JSON at intervals.

Based on the ws-rs guide, I’ve made a struct that represents the established connection. The Handler trait lets me respond to messages and all the websocket lifecycle events by implementing the relevant functions.

Since I don’t care about messages from the server I only implement on_open and use it to start a new thread for sending updates. ws-rs takes care of thread safety—all I need to do is clone the Sender and give it to my new thread where it can use it to send messages. For this test I’ve created a hard-coded status and simulated playback by increasing the position on every update.

Similarly to listen, connect blocks for the duration of a complete connection, after which it returns and the program terminates. This disposes of the update thread, which would normally need to be cleaned up properly.

All this makes me very happy. I love using enums in data models and enums with associated values are even better. In my experience translating enums to numbers or string placeholders can be tedious and a great source of bugs. Having them transparently travel through the JSON means I can use them heavily and be guaranteed that they will have valid values at the other end. And if I did want to create a browser interface it would be easy to replicate the format in Javascript, albeit with none of the safety. I think I’m going to carry on with Rust for this project.

In the meantime Swift 4 finally has good support for JSON encoding so I am looking forward to using that at work.

 

Posted in programming, rust | Leave a comment

Operating Packet Radio on Debian

To experiment with packet radio there are some basic things I need to be able to do. I need to monitor all packets, including those not addressed to me. I need to send individual packets, including a particular digipeater path if I choose. I need to make proper AX.25 connections to connect to BBSes. Ideally I want people to be able to make connections to my node too, and act as a digipeater for others to use.

The old hardware TNCs have all this functionality built in and you control it by connecting to its serial port and typing a series of specialised commands in the terminal. When I attach a KISS TNC to Debian, either in hardware or using a software modem, it’s totally not obvious how to use the AX.25 functionality in Linux to do these same basic tasks.

This post is a short overview of the tools built into Debian/Raspbian that I use to do the same things that you can do with a traditional TNC.

The Setup

The required packages on Debian are ax25-apps and ax25-tools.

To use the Debian AX.25 software the modem must be connected as a real network interface. In other words if you run /sbin/ifconfig you should see an interface called ax0:

Broadly speaking the port and callsign need to be defined in /etc/ax25/axports, the TNC needs to be running/connected, and finally kissattach binds it to an interface. The Dire Wolf soundmodem documentation and TNC-Pi documentation both have very clear instructions about how to do this configuration. The details will vary for different hardware.

Monitoring packets

This is my favourite set of parameters which enable timestamps, include outgoing packets in the output, and enable the use of ANSI colours in the output. See the man page for more detail.

It stays running and as packets are sent and received it presents neat output like the following until it is interrupted with ctrl-c.

Sending individual packets and beacons

Individual UI or “unproto” packets are useful for testing, APRS, having conversations via the ISS and so on. There is a terminal program that sends these and it’s called beacon.

To include a digipeater path use a space-separated -d parameter.

In this example CQ is the destination, ARISS is the callsign of the digipeater I want to use, and “1” is the name of my radio port, i.e. the first column in my axports file. This could easily be wrapped inside a shell script to avoiding needing to type so much every time.

The main purpose of beacon is to transmit a message periodically. If you don’t use -s it will go into the background and keep sending messages every 30 minutes, which is another useful function of some TNCs. See the man page for more info.

Connecting to nodes

axcall is an interactive terminal for connecting to a remote node. The top part of the screen shows text coming back from the remote station and the four lines down the bottom are a buffer where you type.

To terminate a connection cleanly from the local end, press ctrl-] to activate the menus then the down arrow twice to select the “Exit” option, then press enter. There are also menu options for sending data files which I haven’t experimented with much.

Note that traditionally AX.25 terminals indicate the end of the line with a single carriage return and this is what axcall uses. This is different from both Windows (CRLF) and Unix (LF) but is the same as pre-OSX Macs.

Receiving packets

I have more experimenting to do here but here’s an easy way to hook up a program to receive incoming connections using ax25d, an AX.25 equivalent of inetd.

Delete or rename /etc/ax25/ax25d.conf to get rid of all the example junk and replace it with something like this:

The ax25d configuration format is complex but this is enough to make it work. Note that the callsign including SSID and the port name from axports are both included in the square brackets.

The path to the program must be followed by all its arguments. If you are familiar with systems programming you will recall that the first argument to a program is always its own name.

Then run sudo ax25d, which will disappear into the background if everything is fine. Now whenever a station connects they will receive a fortune then have the connection promptly terminated.

Showing connection state

This shows the stations that you are currently connected to and whether there is a program listening for ax25 connections.

Example output:

 

 

Posted in radio | Leave a comment

FSQ Experiments in VK7

Several Tasmanian amateur radio operators hang out in a Matrix chat room, #vk7:matrix.org. Last week Ben VK7BEN shared a video about a new digital mode called FSQ (Fast Simple QSO), which was developed by ZL1BPU and ZL2AFP. Now four of us in the Hobart area are busily attempting to replace augment our online chat with a radio equivalent.

On air FSQ looks and sounds like a faster version of JT65, giving you data speeds of a few characters per second while being able to decode signals right down in the noise.

The main difference compared with JT65 or PSK31 is that the protocol is designed for multiple stations sharing a single fixed frequency. A number of high-level features are built in like saving and retrieving telemetry data, transmitting images, and relaying messages via another station. These are implemented in both the dedicated FSQCALL software and in new versions of fldigi.

160×120 image transmission from VK7NTK to VK7IS on 2m FM

Ben and I are physically located in middle of the group, close to the city of Hobart. Ian VK7IS is about 20 km south and Scott VK7LXX about 50 km east.

FSQ is designed for HF and that seems to be where much of the traffic is—in particular on 40 metres at 7.104 MHz (+1.5 kHz USB centre) and to a lesser degree 7.105 MHz. This frequency simply doesn’t work well over short distances and VK7IS and I have a lot of trouble hearing each other.

Over the weekend we’ve been trying 2 metres and this has worked much better. We’ve had success on both SSB and FM but early indications are better results on SSB. The downsides are that it’s much more sensitive to tuning (I ended up having to shift my dial frequency 90 Hz) and you can’t use any old FM radio.

Even with a small vertical antenna and only 5 watts on my FT-817 I can comfortably work both VK7IS and VK7LXX at the fastest speed, FSQ-6. Ian can hear Scott but not the other way around, so it seems we’re pushing our luck with our current setups.

Since I could work both stations this was a good opportunity to try out the relay features. This is built into FSQ and works similarly to AX.25 digipeating—the transmission includes the sending callsign, the destination callsign and the relay station callsign, who will retransmit automatically when they are listening with the FSQ software.

Scott used this to transmit some “telemetry data” via me, which ended up in a file on Ian’s hard drive. This is slightly processed output showing Scott’s original request, my retransmission, then relaying the automatic acknowledgement that the data was stored.

It’s a fun mode with lots of opportunities for experimentation. If you’d like to give it a go either on VHF or HF grab some software and join us on Matrix!

Posted in radio, tasmania | Leave a comment

Introduction to the GNU social API

I’ve been working on an iOS GNU social client for a while and along the way I’ve had to learn a bit about how GNU social’s HTTP API works. I thought I would write up a few things I wish I knew before.

Notice Formats

The first thing to know is that the GS API typically lets you download information in three formats:

  • Twitter-compatible JSON (.json)
  • StatusNet XML (.xml)
  • ActivityStreams XML (.atom)

You specify what kind of data you would like by putting the appropriate suffix on the end. For example these URLs report this status in the different formats:

JSON is pretty easy to use, especially if you’re building a web client. GS has some extra stuff going on but the Twitter API reference is still really helpful here. The metadata is a bit limited and JSON on the whole does not have great information for parsing out things like mentioned tags or groups. Be warned that if you are using a server that has the Qvitter plugin installed (it looks like slightly old-school Twitter), this plugin adds extra data to the JSON format. If you’re relying on a field, make sure it’s present on a server that does not have Qvitter.

As for StatusNet XML, I don’t really understand what niche it fills so I won’t say anything about it. Have a look at it and let me know what it’s for.

Atom ActivityStreams is a defined standard and so far I’ve found it to have pretty good identifiers for linking data and including metadata like tags and attachments. It is missing some basic information like repeat/fave counts, which I am going to add to GS when I stop writing blog posts and do more coding. I would like to write up a fuller description of the Atom later but you will probably understand it fine if you take some time to read through it carefully. The biggest problem is that it’s verbose.

Using curl to receive and send sample data

It’s super helpful to just play around with the API. On Mac/Linux you can use curl on the command line:

curl https://gs.sdf.org/api/statuses/show/1094190.atom

If you would prefer to save it to a file you can do that:

curl -o my_example.atom https://gs.sdf.org/api/statuses/show/1094190.atom

GS supports HTTP basic auth for the API endpoints that require authentication. You can do that in curl straight on the command line. For example to post a status:

curl --user username:password -d "status=Test Post" https://gs.sdf.org/api/statuses/update.json

Paging and updating timelines

When you’re accessing a list of notices you can use the since_id, max_id, page and count parameters to download notices without redownloading, and without accidentally missing any. This is explained super clearly on Twitter’s developer website and the parameters work in exactly the same way on GS. I highly recommend reading it.

Point of interest in the GNU social code

URL router

lib/router.php has lots of sections of code like these:

It shows that that URLs of the form api/gnusocial/config.XXX are accepted, where XXX can be either xml or json, and the PHP component that handles it is called ApiGNUsocialConfig. You can search for this on the command line using a command like git grep ApiGNUsocialConfig inside the repository where you checked out GNU social. My recommendation would be open up the whole GNU social source folder in an IDE like Visual Studio Code, which will automatically give you nice syntax highlighting and project-wide search.

API handlers

Once you find out you’re dealing with an API endpoint “ApiGNUsocialConfig” you will find it in a file named apignusocialconfig.php.

This is the file that’s responsible for building up the result and sending it back to your HTTP client. Often they have very good comments describing the interesting parameters and how it works.

NoticeStream subclasses

If you look at a timeline endpoint like apitimelinehome.php, there’s not a lot of code in it. The process of querying the database for the relevant notices is handled by a subclass of NoticeStream. In this case: InboxNoticeStream.

It is these in these files that the actual SQL queries are constructed, which will help you to understand which notices appear and why.

URLs of interest

To wrap up, here are some interesting API endpoints.

Showing a single notice (ApiStatusesShowAction)
https://gs.sdf.org/api/statuses/show/1094190.json

Posting a notice (ApiStatusUpdateAction) (authenticated)
https://gs.sdf.org/api/statuses/update.json

Home timeline (ApiTimelineHomeAction) (authenticated)
https://gs.sdf.org/api/statuses/home_timeline.atom?page=1&count=50

Public timeline (ApiTimelinePublicAction)
https://gs.sdf.org/api/statuses/home_timeline.atom?page=2&count=10&since_id=1094190

User timeline (ApiTimelineUser)
https://gs.sdf.org/api/statuses/user_timeline.json?id=thomask

Obtaining server settings like name and character limit (ApiGNUsocialConfigAction)
https://gs.sdf.org/api/gnusocial/config.json

Checking if a username and password are correct (ApiAccountVerifyCredentialsAction) (authenticated using credentials under test)
https://gs.sdf.org/api/account/verify_credentials.json

Posted in free software, internet, programming | Leave a comment