Home e-mail: laurie@tratt.net   twitter: laurencetratt   twitter: laurencetratt
email updates:
  |   RSS feed

Programming Style Influences

May 10 2022

Blog archive

 
Last 10 blog posts
Programming Style Influences
snare: a Minimalistic GitHub Webhooks Runner
April Links
Where do Research Problems Come From?
Practising Programming
Making Rust a Better Fit for CHERI and Other Platforms
When is a Blog a Blog?
 
 
Last 10 essays
Static Integer Types
Automatic Video Editing
The Evolution of a Research Paper
Automatic Syntax Error Recovery
Stick or Twist?
Which Parsing Approach?
Alternative Sources of Advice
Minimum Times Tend to Mislead When Benchmarking
A Quick Look at Trait Objects in Rust
Why Aren’t More Users More Happy With Our VMs? Part 2
 
If you ever talk to, or read an interview with, a musician, they will inevitably, and quickly, end up talking about their influences. While there is an element of nostalgia to this, perhaps even of acknowledging debts, I see its main purpose as helping spread knowledge about who the most interesting set of musicians are.

In programming, in contrast, we rarely talk about our influences, other than a frequently expressed allegiance to a single programming language. This seems a shame to me, because it denies new people to our field helpful pointers to programmers and systems whose style might be a useful influence.

As a modest attempt to rectify this situation, I'm going to show how one particular system, OpenBSD, has had a big influence on my programming style over time. It's far from the only system that's influenced me – not to mention various programmers and communities who've also been an influence [1] – but I need something concrete to use as an example.

After a long period as a "normal" user, over time I found myself diving more often into OpenBSD's code. Mostly, I was simply trying to understand how something worked, though occasionally I was trying to fix something [2]. However, the C code that makes up the bulk of OpenBSD seemed alien to me, because it was incredibly terse, and I often gave up trying to understand it rather quickly. In contrast, the programming style that I was using was very verbose. I used long variable names (on the basis that they helped readers understand what was going on), lots of whitespace (on the basis that it helped readers understand what parts of a file were most related to each other), and wasn't afraid to duplicate code with minor variations (on the basis that each variation was then clearer to read).

You can perhaps imagine how surprised I was when, for example, I looked at the source code for OpenBSD's echo command [3]:

int
main(int argc, char *argv[])
{
  int nflag;

  /* This utility may NOT do getopt(3) option parsing. */
  if (*++argv && !strcmp(*argv, "-n")) {
    ++argv;
    nflag = 1;
  }
  else
    nflag = 0;

  while (*argv) {
    (void)fputs(*argv, stdout);
    if (*++argv)
      putchar(' ');
  }
  if (!nflag)
    putchar('\n');

  return 0;
}
It boggled my mind that a useful command could require so little code. If I'd written that, it would have required at least 2, probably 3 times as much code [4]. What was my conclusion? Well, of course, being a normal human being, I knew that the OpenBSD version of echo was unreadable (I mean, what human can interpret *++argv?!), and that my (hypothetical) longer version would have been much better. So I kept on writing code in my verbose style.

Several years later I came to write extsmail, a Unix email sending daemon — the first time that I'd tried to write such a program. I spent quite a while looking through code in the OpenBSD source tree to try and understand how such a daemon might work. Gradually I started to realise that not only could I cope with OpenBSD's terse code style, but it was actually easier for me to read it than code I'd written myself. This was, to put it mildly, surprising: I thought my programming style made life easier for readers but the polar opposite style seemed to be better.

At the time I couldn't articulate why OpenBSD's style was better, partly because my ego didn't really want to let go of the idea that verbose code was better. I now believe the reason is fairly simple: in essence, OpenBSD's style squeezes more code onto the screen. It makes local chunks of code (e.g. for loops) a bit harder to read but gives the reader a better sense of the wider context (e.g. a function or set of related functions). This, I believe, is a good trade-off because, in programming, the wider the context one is considering, the harder it is to gain a good understanding of that context. My style made understanding local context easier but made it noticeably harder to understand wider contexts.

Even before I could articulate the reasons why OpenBSD's style was better, I started to adjust my programming style. It might not be obvious if you look at parts of extsmail's source code, but I started trimming down variable names and whitespace, and I became more comfortable using what I'd once thought of as "weird" C idioms. Each time I went a little further in that direction, I found my code a little easier to read, especially if there was a gap in time between writing and reading that code.

Over time I realised that a key aspect of this programming style is rewriting. I start by trying to make something which works then I try and make it smaller. By the time I'd come to write hk (the first new C program I'd written in quite some time), I think my evolution was clear. hk's entire source code is currently a paltry 224 lines. If you compare this chunk of code:

unsigned int ignorable_modifiers(Display *dpy) {
  KeyCode ignorable_keycodes[sizeof(IGNORABLE_MODIFIER_KEYSYMS) / sizeof(KeySym)];
  for (size_t i = 0; i < sizeof(IGNORABLE_MODIFIER_KEYSYMS) / sizeof(KeySym); i++) {
    ignorable_keycodes[i] = XKeysymToKeycode(dpy, IGNORABLE_MODIFIER_KEYSYMS[i]);
  }

  XModifierKeymap *mm = XGetModifierMapping(dpy);
  unsigned int mask = 0;
  for (int i = 0; i < 8 * mm->max_keypermod; i++) {
    for (size_t j = 0; j < sizeof(ignorable_keycodes) / sizeof(KeyCode); j++) {
      if (ignorable_keycodes[j] != NoSymbol && mm->modifiermap[i] == ignorable_keycodes[j]) {
        mask |= MODIFIER_MASKS[i / mm->max_keypermod];
      }
    }
  }
  XFreeModifiermap(mm);

  return mask;
}

int main(...) {
  ...
  unsigned int ignore_mask = ignorable_modifiers(dpy);
  unsigned int mask;
  KeyCode keycode;
  parse(dpy, argv[0], &mask, &keycode);
  XGrabKey(dpy, keycode, mask, root, False, GrabModeAsync, GrabModeAsync);
  for (unsigned int i = 0; i <= ignore_mask; i++) {
    if ((ignore_mask & i) == i) {
      XGrabKey(dpy, keycode, mask | i, root, False, GrabModeAsync, GrabModeAsync);
    }
  }
  ...
}
to the approximate equivalent in xbindkeys you'll notice a significant difference in approach. My first attempt at the above looked a lot like xbindkeys: I had to keep rewriting in order to reduce the code until I felt I'd made it sufficiently small while still (hopefully!) being correct and readable [5].

Wrapping up

The point of this post isn't try and persuade you that you should also use OpenBSD's style as an influence, but I wanted to show what the effects of a programming style influence can be. I could equally have talked about how BBC BASIC, Arm 2 assembly, Python, Template Haskell, or many other languages and systems have influenced me, and I think the overall thrust of this post would have been similar.

In my opinion, any given influence can, or at least should, only take you so far: there will always be differences in your context from that of the influence itself. For example, I am much more fond of long comments than OpenBSD is [6] because I find that helps me, and I think others, reason about the code's correctness more thoroughly. In a very different manner, and heavily influenced by my time working in RPython, I have been persuaded of the virtues of having extensive test suites.

An interesting question is when one is in the right place to accept and make use of any given influence: any given influence necessarily comes with a context where that influence makes sense. For example, when I was more of a beginner, I don't think OpenBSD's style could have influenced me: its costs would have been obvious to me, but I doubt I'd have been able to appreciate its benefits.

One good thing about growing older is that I add to my set of influences far more often than I remove something. In fact, I'm not even sure if I've ever removed an influence, though some have been wholly superseded by others [7]. These days most of my programming is in Rust. OpenBSD's C style still influences my Rust style, because the core lessons I took from it apply to any other language. However, new influences have made their impression upon me: for example, I've been heavily influenced by some of the functional programming idioms surrounding iteration that Rust encourages.

One implicit assumption amongst musicians is that there's nothing wrong with singing the praises of those who influenced you. It doesn't make the musician in question an unquestioning "fanboy" or diminish the novel aspects of their music. It would be nice if we could do something similar in programming. Personally, I've been lucky to be influenced by some great software and I hope that my influences will continue to evolve!

Acknowledgements: thanks to Edd Barrett, Lukas Diekmann, Dan Luu, and Davin McCall for comments.

If you’d like updates on new blog posts, follow me on Twitter,
or subscribe to email updates:

Footnotes

[1] It's worth noting explicitly that that beyond those I'm consciously aware of, other programmers and systems will have influenced me, in particular, but not only, when that influence is via a third party.
[2] Over the years I've had a small number of small patches included in OpenBSD. My contributions are insignificant, but it's been my small attempt to pay back a bit of the debt I owe to this wonderful piece of software.
[3] The version I'm quoting is from around the time I'd have first seen the source code. Since then echo has bloated significantly with two extra lines to add pledge support!
[4] Though I doubt I'd have managed to make it quite as long as GNU's version of echo, even if I had implemented the same number of features.
[5] If you're wondering why there are some very long variable names in this chunk of code, it's because I kept getting confused by X11's separation of key symbols and key codes. In fact, looking at the code now, I've again forgotten which is which!
[6] As a couple of random examples see this file from snare or this file from grmtools.
[7] For example, in my teens I used a language, whose name I can no longer even recall, created by someone on the bulletin board I used. It was, roughly speaking, a mix of Pascal and C. I learnt quite a bit from that, although I think I (re)derived the same lessons when I got further into C in later years.
It's worth noting explicitly that that beyond those I'm consciously aware of, other programmers and systems will have influenced me, in particular, but not only, when that influence is via a third party.
Over the years I've had a small number of small patches included in OpenBSD. My contributions are insignificant, but it's been my small attempt to pay back a bit of the debt I owe to this wonderful piece of software.
The version I'm quoting is from around the time I'd have first seen the source code. Since then echo has bloated significantly with two extra lines to add pledge support!
Though I doubt I'd have managed to make it quite as long as GNU's version of echo, even if I had implemented the same number of features.
If you're wondering why there are some very long variable names in this chunk of code, it's because I kept getting confused by X11's separation of key symbols and key codes. In fact, looking at the code now, I've again forgotten which is which!
As a couple of random examples see this file from snare or this file from grmtools.
For example, in my teens I used a language, whose name I can no longer even recall, created by someone on the bulletin board I used. It was, roughly speaking, a mix of Pascal and C. I learnt quite a bit from that, although I think I (re)derived the same lessons when I got further into C in later years.

snare: a Minimalistic GitHub Webhooks Runner

May 4 2022

Starting with extsmail I seem to have developed the accidental hobby of occasionally developing new tools in the Unix tradition. In this post I'm going to introduce snare, a minimalistic GitHub webhooks runner daemon for those cases where something like GitHub actions don't give you sufficient control. I've been using snare in production for over 2 years for tasks from running CI using specific hardware to sending out email diffs to automatically restarting Unix daemons when their configuration has been changed. It's a simple, but now stable, tool [1].

What is a webhooks runner? Well, whenever something happens to a GitHub repository – for example, a new PR is raised, or a commit is pushed to a PR – GitHub can send an HTTP request to a specified URL informing it of what happened. Configuring a webhook for a given GitHub repository is relatively simple: go to that repository, then Settings > Webhooks > Add webhook.

snare is a simple program which listens for GitHub webhook events and runs Unix shell commands based on them. You need a server of your own to run snare on. When you add a GitHub webhook, you then need to specify http://yourmachine.com:port/ [2], a secret (in essence, a shared password between GitHub and snare) and then choose which events you wish GitHub to deliver. For example, the default "Just the push event" works well if you want to send email diffs whenever someone pushes commits to a repository.

snare needs a configuration file to tell it what to do when an event comes in. A very simple snare.conf file looks as follows [3]:

listen = "<ip-address>:<port>";

github {
  match ".*" {
    cmd = "somecmd";
    secret = "<secret>";
  }
}
In essence, snare will listen on <ip-address>:<port> for webhook events, verifying that they were created with the secret <secret>. Each request is relative to a repository: match blocks match against a "owner/repository" string using Rust's regex crate for regular expressions". Thus ".*" matches against any repository and the Unix shell command somecmd will be run when (any) event is received for that repository.

Let's imagine we want to send out email diffs when someone pushes to the repositories "owner/repo1" or "owner/repo2". We might create a github block along the lines of the following:

github {
  match "owner/repo[12]" {
    cmd = "ghemaildiff %o %r %e %j email@example.com";
    secret = "<secret>";
  }
}
This only matches against the particular repositories we wanted to match. The command we're now going to execute is called ghemaildiff (I'll show an example of this below) and it takes five or more arguments: the repository's owner (%o), name (%r), the GitHub event type (%e), a path to the full JSON of the GitHub event (%j), and one or more email addresses to send diffs to. As you've probably guessed, snare searches for text like %e and replaces it with other text; %% escapes percentage characters, should you need to do so.

One of the big problems when executing commands like this is when something goes wrong -- it's easy for the error to sit unnoticed in a log. Instead, snare allows one to add an errorcmd, which is very similar to cmd, except a) it's only executed when cmd fails b) it has an additional %s modifier, which is a path to a file with the stdout / stderr of the failed command. I typically use it as follows:

github {
  match "user/repo[12]" {
    cmd = "ghemaildiff %o %r %e %j email@example.com";
    errorcmd = "cat %s | mailx -s \"snare error: github.com/%o/%r\" email@example.com";
  }
}
so that if executing a command fails, I'm sent an email that helps me debug the problem.

Security

For most purposes, the example configuration above is enough to use snare in anger. However, any program which takes input from a network and runs commands based on it is a security risk. snare tries to reduce these worries by rejecting incoming requests if any part of the input isn't exactly as expected. The % escape sequences available to cmd are guaranteed to:

  1. satisfy the regular expression [a-zA-Z0-9._-]+
  2. not to be the strings "." or "..".

This means that the escape sequences are safe to use as shell arguments and/or to be included in file system paths.

However, the user still has to be thoughtful in the commands they run which boils down to:

  • All input (including JSON files) must be treated as potentially suspect: I urge you to accept input only if it precisely matches the format you expect, rather than merely rejecting input if it does something that you happen to recognise as unexpected or bad. The problem with the latter approach is that it's easy to overlook things that will subsequently turn out to be bad. Put another way: it's better to be overstrict and relax later.
  • Use at least set -euf (and perhaps more) in shell scripts so that errors in subcommands cause your script to immediately terminate rather than limp on in a way that you almost certainly didn't anticipate.
  • Think carefully about who can cause an event to be triggered: for example, if you run webhooks when a pull request is merged, can someone outside your organisation cause a merge to occur?
  • If a command fails, think about whether your errorcmd (if you have one) can unintentionally leak private information.

I am deliberately making the above scary sounding, because I want to emphasise that you need to use snare in "don't trust until proven trustworthy" mode. If you do so, I believe that snare can be used in a way that is wholly secure.

An example command

You can execute whatever command you want with snare, but here's an example ghemaildiff script which creates simple, but useful, diffs which it sends via email:
#! /bin/sh

set -euf

if [ $# -lt 5 ]; then
    echo "Usage: ghemaildiff <owner> <repository> <event> </path/to/JSON> <email_1> [...<email_n>]" > /dev/stderr
    exit 1
fi

# We only generate diffs for push events
if [ "$3" != "push" ]; then
    exit 0
fi

before_hash=`jq .before "$4" | tr -d '\"'`
after_hash=`jq .after "$4" | tr -d '\"'`
echo "$before_hash" | grep -E "^[a-fA-F0-9]+$" 2>&1 > /dev/null
echo "$after_hash" | grep -E "^[a-fA-F0-9]+$" 2<&1 > /dev/null

owner=$1
repo=$2
shift ; shift ; shift ; shift

git clone https://github.com/$owner/$repo repo
cd repo
for email in $@; do
    git log --reverse -p "$before_hash..$after_hash" | mail -s "Push to $owner/$repo" "$email"
done
Notice that this script doesn't check inputs which snare has already validated (e.g. $1 is snare's %o and has thus already been validated as a sensible input) but is careful to check that the git commit IDs extracted via jq satisfy a very narrow regular expression before passing them on as shell arguments.

Advanced configuration

As you can see from the snare.conf man page, snare doesn't have a huge number of configuration options. That's deliberate, because I wanted to keep snare simple: snare doesn't even provide a builtin way to fetch a repository! However, there are two additional configuration tricks that are worth knowing about.

When a request comes in, snare "executes" all the match statements in the config file, from top to bottom: later settings override earlier settings [4]. This allows the user to set, or override, defaults in a predictable manner. Indeed, snare inserts an implicit match block before the user's configuration:

match ".*" {
  queue = sequential;
  timeout = 3600;
}
I'll explain queue shortly; the timeout is 1 hour. If, for example, the user has this configuration file:
github {
  match ".*" {
    cmd = "somecmd";
    errorcmd = "cat %s | mailx -s \"snare error: github.com/%o/%r\" abc@def.com";
    secret = "sec";
  }
  match "a/b" {
    errorcmd = "lpr %s";
  }
}
then the following repositories will have these settings:
a/b:
  queue = sequential
  timeout = 3600
  cmd = "somecmd";
  errorcmd = "lpr %s";
  secret = "sec"
c/d:
  queue = sequential
  timeout = 3600
  cmd = "somecmd";
  errorcmd = "cat %s | mailx -s \"snare error: github.com/%o/%r\" abc@def.com";
  secret = "sec"
You can override settings as many times as you want in a file: it's a powerful technique!

By default, snare queues requests for any given repository and only executes the next in the queue when the previous command has finished. This is a safe default, but can lead to undue work and delay, particularly for repositories with significant activity. There are two other queue modes. queue = parallel executes requests in parallel to each other. I've not used this much myself, but there are obvious use cases for it.

In contrast, I use queue = evict extensively: it means a repository has a maximum queue length of 1, with any new request coming in replacing the existing queue entry (if it exists). For example, we have many webhooks which build documentation for a repository after a pull request is merged. If several pull requests are merged in quick succession (which is common), there's no point waiting to build the documentation for all the pull requests: we might as well only build the documentation relating from the "latest and greatest" merge. Note that evict does not stop any currently running job.

Summary

snare is a niche tool, but I suspect more people could benefit from this niche than currently realise it: certainly, we've found ourselves using snare in more ways than I ever expected.

An obvious example is where we use it to automatically rebuild and release websites when a commit is pushed to a repository. Less obviously, we frequently pair it with bors and buildbot. Sometimes that's because we need to run actions on specific hardware, but there are other simpler uses too. For example, we use it to build grmtools documentation and force push it to a gh-pages branch on every pull request merge: this way the grmtools documentation is always up-to date, but we don't have to share a GitHub access token in the globally visible .buildbot.sh file. I'm sure other people can think of uses for snare which would never have occurred to me!

At a later date, I'll write a short blog post about my experiences about writing snare in Rust.

If you’d like updates on new blog posts, follow me on Twitter,
or subscribe to email updates:

Footnotes

[1] Rust's cargo is, by some distance, the best language package manager I've used and, in my opinion, a significant factor in Rust's success. However, the culture of having many (many!) small dependencies means that it's not possible to take the traditional Unix approach of OS-level packaging to crates. That means that if I want to make sure that snare users have access to the latest security release of a dependency-of-a-dependency, the expectation is that I release a new version of snare. Most recent updates of snare have thus really just been about updating dependencies.
[2] If, as I recommend, you want to put snare behind https you'll need to use a forwarding proxy server. It would be nice if snare could support https directly, perhaps including automatic certificate support to avoid problems with untrusted SSL certificates.
[3] The reason for an explicit github block is because I can imagine snare easily being extended in the future to cope with the webhooks-equivalents for other sites such as GitLab and the like.
[4] I don't know who first came up with this style of config file, but it's certainly become a common idiom in OpenBSD daemons over the years, which is what influenced me.
Rust's cargo is, by some distance, the best language package manager I've used and, in my opinion, a significant factor in Rust's success. However, the culture of having many (many!) small dependencies means that it's not possible to take the traditional Unix approach of OS-level packaging to crates. That means that if I want to make sure that snare users have access to the latest security release of a dependency-of-a-dependency, the expectation is that I release a new version of snare. Most recent updates of snare have thus really just been about updating dependencies.
If, as I recommend, you want to put snare behind https you'll need to use a forwarding proxy server. It would be nice if snare could support https directly, perhaps including automatic certificate support to avoid problems with untrusted SSL certificates.
The reason for an explicit github block is because I can imagine snare easily being extended in the future to cope with the webhooks-equivalents for other sites such as GitLab and the like.
I don't know who first came up with this style of config file, but it's certainly become a common idiom in OpenBSD daemons over the years, which is what influenced me.

April Links

April 30 2022

If you’d like updates on new blog posts, follow me on Twitter,
or subscribe to email updates:

Where do Research Problems Come From?

April 28 2022

"Research" is a much broader term than most of us who consider ourselves to be researchers realise. My occasional interactions with people in very different disciplines (e.g. biology) have tended to involve a series of culture shocks to both parties. Even within computing, there are significant differences between sub-disciplines. These seem to mostly go unnoticed or, at least, uncommented on, which is a pity: at the very least it's worth knowing that the way we do things isn't the only possible way. One of the most surprising differences, at least to me, is where people expect to find the problems that they then work on.

What is a research problem?

Before I go further, it's worth defining what I mean by "research problem". At the most basic, I mean "the problem I'm trying to solve" or "the thing I'm trying to make better". Research problems tend to come at multiple levels. For example, at the highest level, I want to make software better, but that's such a vague aim that it's difficult to convert into meaningful action. What I have to do is find lower-level, more concrete, problems where I can better define what direction I want to head in and what success might look like. For example, I might say "programs written in programming language X are so slow that programmers twiddle their thumbs waiting for programs to run; if I can make programs in language X run twice as fast as they currently do, programmer productivity will increase".

I've actually done two separate things in that small example. Not only have I defined a problem (programs in programming language X run too slowly) but I've also motivated why that problem is worth trying to solve (it's degrading programmer productivity). The requirement to have such a motivation is a common expectation in the parts of computing I work in, but not everywhere. For example, in some areas of mathematics, it's considered worthwhile to solve a problem irrespective of any possible real-world use. In A Mathematican's Apology, G. H. Hardy said [1]:
I have never done anything 'useful'. No discovery of mine has made, or is likely to make, directly or indirectly, for good or ill, the least difference to the amenity of the world. I have helped to train other mathematicians, but mathematicians of the same kind as myself, and their work has been, so far at any rate as I have helped them to it, as useless as my own. Judged by all practical standards, the value of my mathematical life is nil; and outside mathematics it is trivial anyhow.
Personally, I like the idea that the research problems I'm working on might lead to practical use, though I'm realistic that this won't always work out — after all, I don't have perfect insight into the future.

How I generally find research problems

My most common approach to finding research problems derives accidentally from another habit of mine. I spend a considerable portion of my time in what I think of as "scouting mode" where, for the research problems I'm currently working on, I'm trying to understand what might come next. It's easiest to explain this as involving two steps.

First, I explicitly try to better understand the problem I'm currently tackling, because as I work, on it my impression of exactly what the problem is, or should be, evolves. Mostly I try to think of where the boundaries of the problem are and whether there might plausibly be path(s) to get to those boundaries. This helps me identify both challenges (i.e. "this is going to take longer than I thought") and opportunities (i.e. "this also accidentally solves another problem") that I'm likely to encounter.

Since that first step frequently identifies more challenges than I hoped for (I am an optimist after all), my second step is to look for inspiration, either as tools or techniques, that might help solve those challenges. I take all sorts of things into considerations: blog posts, research papers, conversations, source code, README files (yes, really!), my own experiments, and so on. I guess that I probably average 30-60 minutes a day looking for such inspiration and trying to think through the consequences of what I've found.

Mostly my hunt for inspiration is reactive: I have just stumbled across a problem that I can't fix with a 30 second web search and I need to find a solution for it now. Most of these cases tend to involve relatively minor software details and are often resolved by simply spending a bit more time searching (which might, for example, uncover a new-to-me library that does what I need). Sometimes I realise I've hit something deeper and then I might, for example, spend a couple of hours reading through relevant research literature hoping to find a solution.

However I deliberately spend some time proactively looking for inspiration without a concrete problem in mind: I'm trying to build a cache of tools and techniques that will help me for what I might encounter in the future. My main aim here is to fill a short-term cache for the problems I think I will encounter in the coming weeks and months. This is deliberately unstructured: after all, I don't really know what problems I might encounter even tomorrow! As a pleasant side-bonus this also inevitably populates a long-term cache: it broadens my outlook and gives me a wider sense of what's possible.

Whether I'm reactively or proactively looking for inspiration, occasionally, and seemingly out of nowhere, something will jump out at me as particularly interesting. Sometimes, I realise that an existing set of solutions to a problem are incomplete. For example, when I looked into parsing error recovery, I quickly realised that most existing techniques had been designed with the severe limitations of 1980s hardware in mind. It didn't take much extrapolation to realise that modern hardware might change what was possible. Sometimes, I realise that a new-to-me technique or tool could solve a problem I hadn't even thought of before. Within a few hours of stumbling across meta-tracing, for example, I realised that fairly fast programming languages no longer had to be the preserve of wealthy organisations [2]. A little while later (prompted by an off-hand comment from someone else) I realised that meta-tracing could be the basis of a solution to migrating software between programming languages, a problem I had never even thought of before! Similarly, CHERI opened my eyes to all sorts of possibilities for pragmatically improving the security of existing software, the consequences of which I'm still trying to think through.

Many readers have probably read of Richard Feynman and Richard Hamming's description of problem solving. In Hamming's words [3]:
Most great scientists know many important problems. They have something between 10 and 20 important problems for which they are looking for an attack. And when they see a new idea come up, one hears them say "Well that bears on this problem." They drop all the other things and get after it... Now of course lots of times it doesn't work out, but you don't have to hit many of them to do some great science. It's kind of easy. One of the chief tricks is to live a long time!
How does this relate to my description? Well, when I'm in proactive mode, I don't really have a long-standing list of concrete problems in my head: I'm less trying to match problems and solutions than I am trying to better understand possible problem areas [4]. When I come across a new tool or technique that seems plausible [5] I try to guess what its knock-on effects will be: mostly I'm unable to do so; sometimes I can guess a bit but nothing seems particularly interesting; but, once in a while, whole areas seem to unveil themselves to me. When the latter occurs, I know I've found a research problem I want to work on [6]! Crucially, the problem areas that occur to me are nearly always the result of the proactive scouting I've performed in the past: for me, at least, it's a virtuous cycle of solving current problems and identifying future ones to tackle!

Some other ways of finding research problems

You can probably imagine my astonishment when I realised that in some communities there is a widely accepted list of "the next problems to work upon". For example, in some (many? all? I don't know!) branches of telecoms, where people are often working towards things like the next mobile network communication standards (4G, 5G, etc.), there seems to be a shared understanding based on past experience of what might work and what might not. Propose, for example, the use of a conductor material that didn't work in the previous generation, and you're going to get very short shrift from nearly everyone. Rather, the community seems to have a sense of "there are ten plausible materials, three are currently being trialled, so pick one of the other seven to work on." I am, of course, simplifying and caricaturing what goes on, so please don't see this as a criticism: that community has been highly successful in rolling out a sequence of impressive real-world advances.

A very different way of obtaining research problems is when a field goes through alternating periods of "big ideas" followed by "incremental gains". In computing, the most recent big idea is machine learning. There are now tens of thousands of researchers (at a low-end estimate) taking the basic idea of machine learning and either applying small tweaks to it, or finding new problem domains to apply it to. It can seem like there's an almost infinite set of small problems to work on in machine learning, which means that no-one needs to do too much thinking before choosing one and charging forwards. It's easy to dismiss the individual value of most of this work as very low — yet, collectively, it's clearly pushing this field forwards. This is very different from, say, programming languages where it's close to an expectation that each and every researcher will have a distinct niche that they're working on.

The final approach I've encountered is where research problems come from an external body. For example, if you work in medical research, the priorities of funders will often point you towards working on certain health problems over others. One advantage of this is that the external body can not only drive a critical mass of researchers towards solving a common goal, but help ensure that different researchers don't unnecessarily duplicate work. A disadvantage is that if the external body fixates on a pointless or unsolvable problem, it can cause an awful lot of wasted research.

Fundamentally, I don't think that any of the approaches I've outlined above is inherently better or worse: they're just different. My intuition is that it's a good thing that there are different approaches to finding research problems. And, at the risk of stating the obvious, while finding a good research problem to work on is vital, how one goes about addressing it is an entirely different matter!

Acknowledgements: thanks to Edd Barrett, Lukas Diekmann, and Dan Luu for comments.
If you’d like updates on new blog posts, follow me on Twitter,
or subscribe to email updates:

Footnotes

[1] Time has shown that Hardy was wrong: since his death in 1947, some of his work has turned out to be useful!
[2] For example, at a very conservative estimate, HotSpot – the "main" Java virtual machine – has had somewhere between 1000-4000 person years of effort poured into it. The personnel cost implied by that is staggering — not to mention that the people involved are often an organisation's most talented, who could conceivably be earning the organisation copious income if deployed elsewhere.
[3] As far as I know, we only have a second hand quote of Feynman's advice via Gian-Carlo Rota though the final sentence does sound like vintage Feynman:
You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: 'How did he do it? He must be a genius!'"
[4] Whether this is because I'm too stupid to identify concrete problems in advance and or because areas such as software problems are inevitably woolier than in the physical sciences I'm unsure.
[5] There's no point thinking about the implications of something that I don't think can work.
[6] Conversely, I often fail to get excited by solutions that seem to excite other people, because I cannot see how they project backwards onto research problems. For example, an astonishing number of people I've come across tout WebAssembly as a solution to their problems (and a wide variety of problems at that!). I see it as a neat solution to a niche problem (running C/C++-ish code in a browser) but I am unable to see it as a general solution to other problems. I hope, however, that other people are right and I'm wrong!
Time has shown that Hardy was wrong: since his death in 1947, some of his work has turned out to be useful!
For example, at a very conservative estimate, HotSpot – the "main" Java virtual machine – has had somewhere between 1000-4000 person years of effort poured into it. The personnel cost implied by that is staggering — not to mention that the people involved are often an organisation's most talented, who could conceivably be earning the organisation copious income if deployed elsewhere.
As far as I know, we only have a second hand quote of Feynman's advice via Gian-Carlo Rota though the final sentence does sound like vintage Feynman:
You have to keep a dozen of your favorite problems constantly present in your mind, although by and large they will lay in a dormant state. Every time you hear or read a new trick or a new result, test it against each of your twelve problems to see whether it helps. Every once in a while there will be a hit, and people will say: 'How did he do it? He must be a genius!'"
Whether this is because I'm too stupid to identify concrete problems in advance and or because areas such as software problems are inevitably woolier than in the physical sciences I'm unsure.
There's no point thinking about the implications of something that I don't think can work.
Conversely, I often fail to get excited by solutions that seem to excite other people, because I cannot see how they project backwards onto research problems. For example, an astonishing number of people I've come across tout WebAssembly as a solution to their problems (and a wide variety of problems at that!). I see it as a neat solution to a niche problem (running C/C++-ish code in a browser) but I am unable to see it as a general solution to other problems. I hope, however, that other people are right and I'm wrong!

Practising Programming

April 20 2022

When we see a world-class musician flawlessly play challenging music, it can be tempting to imagine that they were always able to do so. A moment's thought makes it obvious that they must have had to spend huge amounts of time practising basic techniques in order to reach that level. What's less obvious is that they have to continue spending considerable amounts of time simply to maintain that technique, let alone expand it.

In contrast, in programming, we have only a haphazard notion of how one should go about obtaining sufficient technique to become good enough to write good software; and we have almost no notion of continued practise to maintain or expand that technique. Often overtaken by life events – notably, though not only, the requirement to earn money to support a family – many programmers implicitly finish the bulk of their learning early on, and some stop learning entirely.

As someone of rather limited talent, I have long been aware that if I'm not proactive about learning, I will fall even further behind where I would like to be. While some people enjoy practising programming with small puzzles (the programming equivalent of musical scales perhaps?), I am too grumpy to find them enlightening or enjoyable. Instead, I've found that real-world tasks that require me to do something new are where I most improve, and most enjoy improving, my programming skills. Fortunately, and at first accidentally, I've found that it's relatively easy for me to continue practising programming on real-world tasks as part of my day-to-day life.

One habit I have is to try to automate most tasks that look automatable. Especially when I was younger, people would laugh at this tendency: "you've spent 4 hours to automate what you could have done in 2." They were nearly always right in the short term, but what's astonishing is how often, with only minor variations, the same tasks crop up repeatedly [1]. If that same 2 hour task crops up twice, I have broken even, and if it comes up three times, I've come out ahead.

When I started trying to automate things, I was pretty bad at doing so: I was notably less competent than my peers. But just as a musician gains compound interest from practising multiple hours a day, so did I. Each task I automated not only improved my basic programming technique, but also opened my eyes to new techniques and tools that might help me in future tasks. For example, a lot of the early things I automated boiled down to text processing: parsing input, transforming it, and outputting it in a different form. I didn't know that there was a theory of parsing (let alone parsing tools!) but, gradually, I built up an understanding of reasonable ways to go about this kind of task. Something similar happened for other areas such as GUI and systems programming. Broadly speaking, in my teens and early 20s I was often running a net loss in terms of time; I'd broken even by my mid to late 20s; and by the time I was 30 it was very clear to me that my investments had paid off. Those more talented than I would probably have reached this point well before that time, but better late than never!

These days, mostly because of the time that other responsibilities require, I am more careful about which tasks I'm willing to invest time in automating. Experience has given me a better sense of what tasks are plausibly automatable and what tasks I might learn something more general from. I also have a wider palate of tools to call upon: for example, I now use clever tricks in my text editor [2] to perform some text processing tasks that I once would have written a program for. However, if a task looks automatable, seems like it might allow me to learn something more general, and isn't covered by other tools, then I'm now more willing than I used to be to invest considerable time in it.

The first time I did that was for extsmaild. I was spending a lot of time on trains and planes and reliably sending email was a problem [3]. I then realised that this might be an opportunity to understand how to write a "proper" Unix daemon in C. It soon became clear to me that the major challenge in a daemon is how to keep going in the face of undesirable situations (including, but not only, outright errors): these days, extsmaild can recover from nearly anything except a lack of memory (which, given its frugality with resources, is an unlikely scenario). I had no idea how many different undesirable situations such a program could encounter [4] and consequently I have learnt a considerable amount from writing and maintaining extsmaild.

When, more recently, I needed a way of executing commands based on actions on GitHub PRs, I felt I wouldn't learn much by writing another C daemon. Instead, I wrote snare in Rust, because I wanted to understand if Rust daemons could be as robust as their C counterparts. Broadly speaking my conclusion was "yes". I also wanted to learn about async/await, a technique I'd never previously used. snare incorporates both an HTTP server (to listen to GitHub events) and a job runner (which waits upon child processes, collecting their stdout/stderr output). I started with the HTTP server, but I found async/await to be a frustrating experience: even in a very small chunk of code, its tentacles spread more widely than I expected, and I found the resulting control flow hard to reason about. When I got to the job runner I reverted back to traditional Unix polling. poll is a horrible interface [5] that's difficult to use correctly, but for me it has the virtue that it isolates the horrors to one place. That wasn't the result from practising that I expected, but it's useful nonetheless.

When I wanted to add a newsletter subscription to my website, I was briefly tempted to use an external service, before realising that I didn't want to worry about the resulting privacy issues. Based on my snare experience, I felt that writing a Rust daemon to handle the backend tasks of a newsletter system would be relatively easy, but I had never written any JavaScript in a webpage: small though it was, the JavaScript that makes the subscription system more user-friendly taught me quite a bit. When I came to add a comments system to my blog, I was able to be slightly more ambitious in what I expected of JavaScript.

As that last example might suggest, some of my programming practise is smaller scale, though still nearly always on real-world code. For example, sometimes I will try a new, or different, library to see if it helps me solve a familiar task quicker or better. Sometimes I might look at a function I've written and try to rewrite it to make it easier to understand, perhaps using a language feature I rarely use. The time I'm investing is small, and the result of such practise is much more likely to be a "failure" than in the bigger cases, but there are successes and, small though they are, they accumulate over time. The most noticeable change, at least to me, is that I've gradually become better at writing code that is concise while, I hope, still being relatively easy to understand.

For me, at least, practising programming is something I do frequently. Fortunately I'm able to incorporate it into my day-to-day life, and fortunately I enjoy the process! Do you deliberately practise your programming skills? If so, how do you go about it? Feel free to comment below!

Acknowledgements: thanks to Edd Barrett, Lukas Diekmann, and Dan Luu for comments.

If you’d like updates on new blog posts, follow me on Twitter,
or subscribe to email updates:

Footnotes

[1] Perhaps because of my job, a lot of tasks happen once a year. One consequence of that is that I sometimes have to do a bit of searching to see where I left the program(s) that automated the task in the previous year!
[2] I have made far more use out of macros and rectangular blocks (sometimes called "multi cursors", though that term also is used differently elsewhere) than I would ever have thought possible.
[3] There have been many advantages to using mutt, and now neomutt, for over 20 years — but also some disadvantages.
[4] If anything, I undersold the lessons I learnt from this experience when I first wrote about them. For example, I had yet to encounter the concept of zombie child processes — a particular eye opener!
[5] Richard Kettlewell's collection of platform quirks around the question "when does a polled file descriptor reach EOF?" is required reading.
Perhaps because of my job, a lot of tasks happen once a year. One consequence of that is that I sometimes have to do a bit of searching to see where I left the program(s) that automated the task in the previous year!
I have made far more use out of macros and rectangular blocks (sometimes called "multi cursors", though that term also is used differently elsewhere) than I would ever have thought possible.
There have been many advantages to using mutt, and now neomutt, for over 20 years — but also some disadvantages.
If anything, I undersold the lessons I learnt from this experience when I first wrote about them. For example, I had yet to encounter the concept of zombie child processes — a particular eye opener!
Richard Kettlewell's collection of platform quirks around the question "when does a polled file descriptor reach EOF?" is required reading.
Blog archive
Home e-mail: laurie@tratt.net   twitter: laurencetratt twitter: laurencetratt