FSQ Experiments in VK7

Several Tasmanian amateur radio operators hang out in a Matrix chat room, #vk7:matrix.org. Last week Ben VK7BEN shared a video about a new digital mode called FSQ (Fast Simple QSO), which was developed by ZL1BPU and ZL2AFP. Now four of us in the Hobart area are busily attempting to replace augment our online chat with a radio equivalent.

On air FSQ looks and sounds like a faster version of JT65, giving you data speeds of a few characters per second while being able to decode signals right down in the noise.

The main difference compared with JT65 or PSK31 is that the protocol is designed for multiple stations sharing a single fixed frequency. A number of high-level features are built in like saving and retrieving telemetry data, transmitting images, and relaying messages via another station. These are implemented in both the dedicated FSQCALL software and in new versions of fldigi.

160×120 image transmission from VK7NTK to VK7IS on 2m FM

Ben and I are physically located in middle of the group, close to the city of Hobart. Ian VK7IS is about 20 km south and Scott VK7LXX about 50 km east.

FSQ is designed for HF and that seems to be where much of the traffic is—in particular on 40 metres at 7.104 MHz (+1.5 kHz USB centre) and to a lesser degree 7.105 MHz. This frequency simply doesn’t work well over short distances and VK7IS and I have a lot of trouble hearing each other.

Over the weekend we’ve been trying 2 metres and this has worked much better. We’ve had success on both SSB and FM but early indications are better results on SSB. The downsides are that it’s much more sensitive to tuning (I ended up having to shift my dial frequency 90 Hz) and you can’t use any old FM radio.

Even with a small vertical antenna and only 5 watts on my FT-817 I can comfortably work both VK7IS and VK7LXX at the fastest speed, FSQ-6. Ian can hear Scott but not the other way around, so it seems we’re pushing our luck with our current setups.

Since I could work both stations this was a good opportunity to try out the relay features. This is built into FSQ and works similarly to AX.25 digipeating—the transmission includes the sending callsign, the destination callsign and the relay station callsign, who will retransmit automatically when they are listening with the FSQ software.

Scott used this to transmit some “telemetry data” via me, which ended up in a file on Ian’s hard drive. This is slightly processed output showing Scott’s original request, my retransmission, then relaying the automatic acknowledgement that the data was stored.

It’s a fun mode with lots of opportunities for experimentation. If you’d like to give it a go either on VHF or HF grab some software and join us on Matrix!

Posted in radio, tasmania | Leave a comment

Introduction to the GNU social API

I’ve been working on an iOS GNU social client for a while and along the way I’ve had to learn a bit about how GNU social’s HTTP API works. I thought I would write up a few things I wish I knew before.

Notice Formats

The first thing to know is that the GS API typically lets you download information in three formats:

  • Twitter-compatible JSON (.json)
  • StatusNet XML (.xml)
  • ActivityStreams XML (.atom)

You specify what kind of data you would like by putting the appropriate suffix on the end. For example these URLs report this status in the different formats:

JSON is pretty easy to use, especially if you’re building a web client. GS has some extra stuff going on but the Twitter API reference is still really helpful here. The metadata is a bit limited and JSON on the whole does not have great information for parsing out things like mentioned tags or groups. Be warned that if you are using a server that has the Qvitter plugin installed (it looks like slightly old-school Twitter), this plugin adds extra data to the JSON format. If you’re relying on a field, make sure it’s present on a server that does not have Qvitter.

As for StatusNet XML, I don’t really understand what niche it fills so I won’t say anything about it. Have a look at it and let me know what it’s for.

Atom ActivityStreams is a defined standard and so far I’ve found it to have pretty good identifiers for linking data and including metadata like tags and attachments. It is missing some basic information like repeat/fave counts, which I am going to add to GS when I stop writing blog posts and do more coding. I would like to write up a fuller description of the Atom later but you will probably understand it fine if you take some time to read through it carefully. The biggest problem is that it’s verbose.

Using curl to receive and send sample data

It’s super helpful to just play around with the API. On Mac/Linux you can use curl on the command line:

curl https://gs.sdf.org/api/statuses/show/1094190.atom

If you would prefer to save it to a file you can do that:

curl -o my_example.atom https://gs.sdf.org/api/statuses/show/1094190.atom

GS supports HTTP basic auth for the API endpoints that require authentication. You can do that in curl straight on the command line. For example to post a status:

curl --user username:password -d "status=Test Post" https://gs.sdf.org/api/statuses/update.json

Paging and updating timelines

When you’re accessing a list of notices you can use the since_id, max_id, page and count parameters to download notices without redownloading, and without accidentally missing any. This is explained super clearly on Twitter’s developer website and the parameters work in exactly the same way on GS. I highly recommend reading it.

Point of interest in the GNU social code

URL router

lib/router.php has lots of sections of code like these:

It shows that that URLs of the form api/gnusocial/config.XXX are accepted, where XXX can be either xml or json, and the PHP component that handles it is called ApiGNUsocialConfig. You can search for this on the command line using a command like git grep ApiGNUsocialConfig inside the repository where you checked out GNU social. My recommendation would be open up the whole GNU social source folder in an IDE like Visual Studio Code, which will automatically give you nice syntax highlighting and project-wide search.

API handlers

Once you find out you’re dealing with an API endpoint “ApiGNUsocialConfig” you will find it in a file named apignusocialconfig.php.

This is the file that’s responsible for building up the result and sending it back to your HTTP client. Often they have very good comments describing the interesting parameters and how it works.

NoticeStream subclasses

If you look at a timeline endpoint like apitimelinehome.php, there’s not a lot of code in it. The process of querying the database for the relevant notices is handled by a subclass of NoticeStream. In this case: InboxNoticeStream.

It is these in these files that the actual SQL queries are constructed, which will help you to understand which notices appear and why.

URLs of interest

To wrap up, here are some interesting API endpoints.

Showing a single notice (ApiStatusesShowAction)

Posting a notice (ApiStatusUpdateAction) (authenticated)

Home timeline (ApiTimelineHomeAction) (authenticated)

Public timeline (ApiTimelinePublicAction)

User timeline (ApiTimelineUser)

Obtaining server settings like name and character limit (ApiGNUsocialConfigAction)

Checking if a username and password are correct (ApiAccountVerifyCredentialsAction) (authenticated using credentials under test)

Posted in free software, internet, programming | Leave a comment

Facebook and capitalist newspeak

It is increasingly difficult to draw parallels between Nineteen Eighty-Four and today’s world. This is partly because over the last twenty years we brought much of the surveillance capability into our homes willingly, but mostly because this happened without creating an equivalent totalitarian state. Although we have problems we are far from the experience of pervasive fear and control the book describes. One key difference is that the fictional Oceania was a collectivist regime while in our real-world capitalist system we need a certain range of liberties for private enterprise to operate smoothly. So long as we continue to purchase more electronics the economy grows and the order is maintained. There is no urgent need to re-purpose that electronics for population control and surveillance.

I enjoy Orwell’s idea of newspeak because unlike surveillance technology it still reads like science-fiction. As he describes it:

The purpose of Newspeak was not only to provide a medium of expression for the world-view and mental habits proper to the devotees of IngSoc, but to make all other modes of thought impossible.

This was done partly by the invention of new words, but chiefly by eliminating undesirable words and stripping such words as remained of unorthodox meanings, and so far as possible of all secondary meaning whatever.

The English language has never been static and nowadays memes and emoji twist it into new and original forms at a hectic pace. It feels more dynamic than ever. How could a simplified language ever replace it*, let alone one prescribed by the government? I don’t think it could—not like that.

For as long as I can remember there have been campaigns on Facebook petitioning them to add a “Dislike” button alongside “Like”. I can think of three social situations where you could use it: when expressing sympathy for bad news, when disagreeing, and when being spiteful about somebody’s good news. All are real things that you might want to express. Facebook could have added the dislike button and its intent would be clear in context. But the second two usages imply obviously negative feelings. Facebook came up with a design that prevents you expressing those negative feelings easily.

In early 2016 they released “reactions”, a set of six buttons to replace one: Like, Love, Haha, Wow, Sad, Angry. The cutesy drawn faces have a purpose—they fit when used sincerely, but they really don’t work if you want to be disagreeable. In that case you have to open the comment box and think of something to type. Maybe you will think twice and move on.

Facebook does not want its users to have negative feelings. New posts from friends and family are the content that brings people back every day. Negative feedback means fewer posts. Is it Orwellian to choose not to have a dislike button? Not really. They are certainly removing secondary meanings and excluding undesirable meanings but if you prefer to type something in English you still have the opportunity.

But what if you didn’t have the opportunity to comment? What if you didn’t see the post at all? Any mildly perceptive Facebook user knows that the News Feed is heavily curated by their algorithm and it is strongly influenced by the reactions you leave on other people’s posts. If you don’t interact with someone they disappear and you won’t see them again until they get engaged. If none of the reactions fit and you don’t bother to compose written comments that person will quickly disappear. Negative thought becomes impossible when you’re not confronted with anything to be negative about.

The unambiguous reactions serve Facebook in other ways. They want to know how we feel about each post so they can tune our feed. Perhaps they would like to wake us up with righteous indignation then give us nice things to look at after work. Who knows what experiments they’re running now? The reactions provide that information much more clearly than likes and written comments ever would.

It is important to remember that Facebook’s revenue comes almost entirely from advertising. Anecdotally this most often takes the form of sponsored posts from a company that one of my friends has “Liked.” (I honestly have no idea why my friends do this.) I see the most ads, i.e. accumulate the most ad impressions, when I’m scrolling through my feed quickly. The last thing Facebook wants is for me to get bogged down in some long status and spend ten minutes composing a detailed comment. They want me to look at a post for two seconds, hit that reaction button, then scroll to the next ad.

In other words, it directly helps Facebook’s profitability if I distill whatever complicated thoughts I’m having into one of six emotions and press the button. So they make it easy.

Is it a capitalist version of newspeak, dumbing down my thoughts so I consume and consume more? They could do better but they’re getting damn close.

*I may be an Esperantist but I’m a realistic one.

Posted in internet | 2 Comments

Riot’s magical push notifications in iOS

A couple of months ago I started playing with Matrix, a federated communication/chat system. Rather than make an account on a public homeserver I installed Synapse on my own server. I installed the Riot app on my iPhone and connected directly to my server. (Synapse is the server and Riot is the client produced by the Matrix team.)

Everything worked well. A bit too well: I was getting push notifications in iOS when I received new messages. Push notifications are very handy but it was confusing. I knew that new messages were arriving at my self-hosted homeserver but there is no way that my server would be allowed to fire notifications into Apple’s push notification service.

It’s a big problem for any kind of federated network such as Matrix, GNU social or XMPP. The entire point is that you can have an account on any server while still participating in the network. If you want to support an iOS client it can’t get realtime notifications directly from the chosen server. Push notifications can only be sent from servers run by the app developer.

You can solve this by playing pass-the-parcel: your homeserver generates an event and sends it to the app developer’s server, who passes it on to Apple, who commands your iPhone to vibrate. This solves the problem nicely apart from some privacy issues which I’ll get to shortly.

But this creates a new problem. Imagine you want to write your own Matrix app for iOS, a free and open source one that you write in your free time. Your users will insist on having push notifications because it’s the 21st century. This requires you to run a push notification server. Even if you’re willing to do this it costs real money on an ongoing basis and it may be the only significant cost you have. You can’t just make it an in-app purchase because the App Store Review Guidelines explicitly forbid taking payments to enable Push Notifications.

If you want to recoup your costs you either have to charge for your app, which will reduce the take-up of the network you presumably want to support, or take a punt and hope nice users give you donations. (This doesn’t often work.)

What we really need is some benevolent company who will release a free iOS app and subsidise the cost of notifications for everybody while focusing on making money some other way. It sounds crazy but that seems to be exactly what Riot is doing.

The above diagram is straight from the Matrix specification. They’ve built mitigation for the push notification problem right into the design.

I have Riot installed on my iPhone and iPad. When I log on to the Riot web interface it shows that the installed apps have registered themselves as “Notification Targets” on my server.

Looking in the pushers table in my server database I can see that they have registered an HTTP pusher with the following URL:

This means when I need to receive a notification my homeserver will make an HTTP POST to that URL with a JSON message describing the content of the notification. The Matrix/Riot servers will take over from there and forward it to Apple.

So that Apple knows where to send it the message contains a pushkey. This comes from the iOS device itself. The app registers for push notifications and obtains an APNS Device Token. MatrixKit then takes this token and uploads it to the homeserver when it creates the HTTP pusher.

So that explains how my push notifications magically work. It’s pretty awesome. The only downside is a small loss of privacy. One of the nice things about federated networks is that you can take total control of your data security. Communications are decentralised, accessible only on the peers that need them. Supporting iOS notifications chips away at that ideal situation, exposing at the very least some metadata to various third parties. I point the blame for that squarely at Apple’s iron grip.

The good news is that it’s easy to limit the amount of information transmitted in the notifications. While the public channels include text snippets when I’m mentioned, if I’m in an encrypted chat it simply says Message from <username>. Apple and Riot may know that I’m talking to a particular person at a particular time (more specifically, that they are talking to me) but not what was said. This is roughly the same privacy you have when you use Signal for iOS. Solution: turn on encryption. According to Matrix’s blog post this will be the default for private rooms in the future.

It’s an interesting situation. On one hand I’m happy to see Riot break the impasse of federated notifications. On the other hand it only solves the problem for Matrix, and relying on the benevolence of a company to keep a technical solution operating doesn’t fill me with confidence. It’s not ideal but at the end of the day it’s considerably better than what you get using Facebook messenger or Slack so at this point I’m happy with the trade-off!

Posted in internet | 3 Comments