pizauth, an OAuth2 token requester daemon, in alpha

One Monday a few weeks back, I found myself unable to send emails via my work account. I had received no advanced warning and there was no clue in the error message I was seeing as to what the cause might be. I was briefly tempted to view my cut-off from work as a good thing before, irritatingly, the sense of duty my parents instilled in me kicked in. After a while I realised that the problem was that my msmtp setup was using basic authorisation – what the rest of us think of as a traditional username and password – to send emails. Not only does my work’s Microsoft Exchange server no longer accept basic authentication, but Microsoft are killing it off for everyone before the end of the year — more people are thus about to experience the same confusion I experienced.

A common alternative to basic authorisation is OAuth2. From the perspective of this post [1], OAuth2 requires you to use a web browser to authenticate yourself (which might then involve two-factor authentication e.g. via your mobile phone) after which you can obtain time-limited access tokens. You can use access tokens to authenticate yourself to, for example, an IMAP or SMTP server. As irritating as I sometimes find two-factor authentication [2], there is no doubt that OAuth2 is more secure overall than basic authentication.

In my case, I solved my immediate problem with email-oauth2-proxy, which has a very cunning approach to the problem (it’s a semi-transparent proxy that provides the illusion of basic authentication to user clients while performing OAuth authentication to users) but it occasionally stalled and required me monitoring an xterm for authentication links which doesn’t sit neatly in my email setup. I then tried mailctl but it segfaulted on me (perhaps because of an incompatibility with OpenBSD) and I’m not familiar enough with Haskell to debug such things.

A few days later I thus found myself writing a “traditional Unix daemon” for obtaining OAuth2 tokens. I ended up with a few requirements (some of these will only make sense at this point in the post if you know OAuth2; if you don’t, I hope these will make sense by the end of the post).

• Asynchronous notification of authentication requests. I do not want to monitor an xterm, or a log file, for authentication URLs.
• Background refreshing. I would like the illusion of immediately available, valid, access tokens whenever possible.
• No secrets stored on disk. Other OAuth2 token requesters do this, and then have to jump through hoops to protect the secrets. Keeping secrets in RAM reduces such problems substantially.

The result of my labours, pizauth, has just had its first alpha 0.1.0 release. For me, “alpha” means “this at least roughly works, but there might be obvious bugs or oversights, and while I hope the user interface won’t change, I will modify it if flaws become apparent.” Probably the main set of people who will be interested in pizauth are those who use traditional Unix email tools (e.g. extsmail, fdm, neomutt, isync, or msmtp) to read or send email. If you are such a person – or even if you’re not! – I would welcome testing on pizauth: does it work with your email providers? with your local software? is the documentation complete and understandable? do you encounter bugs? etc. Here’s what my particular configuration of pizauth looks like when I use neomutt to read an IMAP folder:

Basic setup

If, like me, you have no previous experience of OAuth2, it can be surprisingly hard to envision what it actually entails (see e.g. the OAuth2 site for pointers). In pizauth’s case, we first create a config file at ~/.config/pizauth.conf. For example, let’s assume I want to be able to read and send my work email via Exchange’s IMAP and SMTP servers:

account "work" {
  auth_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize";
  token_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/token";
  client_id = "<your-client-id>";
  client_secret = "<your-client-secret>";
  scopes = [
    "https://outlook.office365.com/IMAP.AccessAsUser.All",
    "https://outlook.office365.com/SMTP.Send",
    "offline_access"
  ];
  login_hint = "<your-email-address>";
}

In essence, OAuth2 requires a URI to authenticate against (auth_uri), a URI to obtain tokens from (token_uri), and the “scope” (perhaps more easily thought of as “permissions”) you want access to (scopes). Although it’s not mandatory, it helps to add login_hint which is your email address.

client_id and client_secret are, in my opinion, blots on OAuth2: they’re best thought of as being a fixed username and password. For unmonitored server applications, having a password makes sense but for end user applications it is simply security by obscurity. If you’re lucky, your OAuth2 service provider will only require client_id and will tell you what a valid value for it is. If you’re moderately lucky, your OAuth2 service provider will provide a mechanism to let you generate unique values for both client_id and client_secret. If you’re unlucky, you’ll have to guess what valid values your OAuth2 service provider will accept — often other people on the internet will have sussed out values that will work. I hope that, in the future, user-facing systems will stop thinking that client_id and client_secret are anything other than a needless barrier to OAuth2 usage.

With configuration complete, we then start pizauth’s server:

$ pizauth server

At this point, absolutely nothing visible will happen: pizauth will daemonise itself, and wait for you to ask it to do something. What you probably want to do is to show the current OAuth2 access token:

$ pizauth show work
ERROR - Token unavailable until authorised with URL https://login.microsoftonline.com/common/oauth2/v2.0/authorize?access_type=offline&code_challenge=xpVa0mDzvR1Ozw5_cWN43DsO-k5_blQNHIzynyPfD3c&code_challenge_method=S256&scope=https%3A%2F%2Foutlook.office365.com%2FIMAP.AccessAsUser.All+https%3A%2F%2Foutlook.office365.com%2FSMTP.Send+offline_access&client_id=<your-client-id>&redirect_uri=http%3A%2F%2Flocalhost%3A14204%2F&response_type=code&state=%25E6%25A0%25EF%2503h6%25BCK&client_secret=<your-client-secret>&login_hint=<your-email-address>

Perhaps unsurprisingly there isn’t a token to show yet. Instead what we’ve got is a (half kilobyte long) URL. If we paste that URL into a web browser, Microsoft will ask us to authenticate ourselves in the normal fashion for your organisation, which might involve two-factor authentication and so on (if you’re already authenticated in that browser, and lucky, you probably won’t have to do anything). When authentication in the browser is complete, you will be sent to a webpage which says simply “pizauth processing authentication: you can safely close this page.” By the time you’ve closed that window, the chances are that pizauth will have completed its background authentication work. With luck, pizauth show will then show an access token [3]:

$ pizauth show officesmtp
DIASSPt7jlcBPTWUUCtXMWtj9TlPC6U3P3aV6C9NYrQyrhZ9L2LhyJKgl5MP7YV4

That access token can then be used to read and send email. So now you need to configure your email software to call pizauth. For example, I configure msmtp as follows:

account work
auth xoauth2
tls on
syslog on
passwordeval pizauth show work
user your-email-address
port 587
host smtp.office365.com

For neomutt and IMAP [4]:

set imap_user = "your-email-address"
set folder = "imaps://outlook.office365.com:993"
set spoolfile = "+INBOX"
set ssl_force_tls = yes
set imap_authenticators="oauthbearer:xoauth2"
set imap_oauth_refresh_command="pizauth show work"

Asynchronous notifications

It can be rather irritating to manually call pizauth show to find out the URL you need to put into your browser, especially as periodically (ranging, typically, from days to months) you will find you need to reauthenticate yourself against the OAuth2 server. Put another way, authentication requests are potentially asynchronous.

pizauth thus allows you to run arbitrary shell commands when authentication is needed via the global auth_notify_cmd setting in pizauth.conf. This could simply open the authentication URL up in your default browser:

auth_notify_cmd = "open \"$PIZAUTH_URL\"";

When pizauth runs the shell command, it sets two environment variables: $PIZAUTH_ACCOUNT is the account name (in this case “work”); and $PIZAUTH_URL is the authentication URL I need to run in my browser.

Personally, I want a pop-up to display on my XFCE desktop via notify-send. Most notification daemons can parse basic HTML, which is great, because I can then click on a link in the notification to go to a browser and authenticate myself. However, XFCE’s notification daemon doesn’t parse ‘&’ characters correctly, so I have to transform them into ‘&’ with sed. That means that I’ve ended up with a slightly scary looking command:

auth_notify_cmd = "notify-send -t 30000 'pizauth authorisation' \"<a href=\\\"`echo $PIZAUTH_URL | sed 's/&/&amp;/g'`\\\">$PIZAUTH_ACCOUNT</a>\"";

The notification stays alive for 30 seconds (-t takes its argument in milliseconds) [5]. If I miss that notification, auth_notify_cmd will be rerun every 15 minutes (the default value for the auth_notify_interval setting).

When and where to start pizauth

pizauth is a traditional background Unix daemon, so you can start it whenever you want and it will happily put itself into the background and stay there. However, where is a good place to call pizauth server from?

Personally I do so in my ~/.xsession file so that pizauth starts when I log into X: that means that it picks up the necessary environment variables for notify-send to work.

However, there’s no reason that you can’t start pizauth’s server elsewhere e.g. in your shell’s login file. The server checks for the existence of a (functioning) pizauth server and won’t start if one exists, so you don’t have to worry about accidentally starting two pizauth servers.

Background refreshing

OAuth2 access tokens typically have a short expiry time (around 1 hour is common). Although largely hidden from users, there are also refresh tokens which allow one to obtain new access tokens (a reasonable intuition is that refresh tokens are a sort of “superuser token”). Refresh tokens typically have a long expiry time (days, weeks, or months). One can use a refresh token to obtain a new access token at any point (even before the access token has expired), but some systems put a limit on how many new access tokens you can obtain in a period of time.

pizauth thus faces an interesting challenge: when should it refresh access tokens? It could wait until the first show request after an access token has expired and then refresh it, but that can cause irritating pauses. It could refresh the access token upon every show request but as well as irritating pauses this could lead to users exceeding the access token limit. Both these approaches also imply that displaying access tokens should block until refreshing has finished. I suspect that blocking in this way will cause users to assume that displaying access tokens always succeeds. That assumption breaks down at the point that refresh tokens expire and reauthentication is required.

The approach I’ve taken is two fold. First, pizauth show is non-blocking. As we saw above, it will immediately return with an error when it is first called; it will also return with an error if it has been unable to refresh an expired access token. Second, pizauth has a background thread which tries to refresh access tokens just before they’re about to expire (by default 90 seconds before, controlled by the per-account refresh_before_expiry setting). In general this means that pizauth show shows access tokens immediately.

However, access tokens can be externally revoked [6], so there’s no guarantee that the access token pizauth displays is valid. pizauth provides two mechanisms to resolve this problem. Users (or programs) that realise an access token is no longer valid can explicitly call pizauth refresh to forcibly refresh an access token: if refreshing fails, this returns an error (with an authentication URL). However, in general, we don’t want users to have to explicitly request refreshing. pizauth thus provides a “backstop” setting refresh_at_least setting (which defaults to 90 minutes) which forcibly refreshes an access token even if hasn’t expired. In practise this means that, at worst, pizauth will show invalid access tokens for a fixed period of time before either obtaining a new access token or notifying the user that they need to reauthenticate.

In summary, pizauth’s default settings mean that users don’t really have to worry about refreshing: it should mostly “just work”.

Config reloading

Many Unix daemons have a way of reloading their config file without having to fully shutdown and restart (often via the dicey SIGHUP signal). Since pizauth does not store any state on disk, shutting down and restarting is particularly irritating, as it would require reauthenticating every account. pizauth thus has a pizauth reload command (which doesn’t use SIGHUP!) which causes pizauth to reload its configuration.

In my experience, configuration reloading is often done in a crude fashion. I soon realised that wasn’t acceptable for pizauth where sloppy configuration reloading could cause access tokens being handed out to the wrong server (e.g. if you swap two accounts’ auth_uris; or delete an old account and create a new one with the same name but otherwise different details). It might not be obvious, but this is a nightmare scenario: if I give an access token intended for server S1 to server S2, then S2 could use that access token to illicitly obtain services from S1. This isn’t very likely to happen, but I wanted pizauth to make sure that users could never shoot themselves in the foot.

When pizauth reloads the user’s configuration, any account which is changed in any way has its tokens removed (i.e. the user will have to reauthenticate from scratch), thus removing the shotgun entirely. Because this is an important security property, pizauth has a carefully crafted internal API which causes it to crash if this property could be violated. I’ll probably write a more detailed post on that at a later point.

Summary

pizauth solves a problem I have, and which I know some other people have too. My aim has been that it is, as far as possible, a “configure once and then leave alone for ever” daemon. If you’re interested in such a program, please try out the alpha release and let me know what problems you find (I am bound to have made mistakes and overlooked things)! For example, I and my set of initial testers have only tested pizauth on Microsoft and Gmail’s OAuth2 servers: it might be the case that other systems need slight tweaks to get them to work. But, hopefully, you will find pizauth a useful tool!

Acknowledgements: thanks to Edd Barrett, Dejice Jacob, and J. Ryan Stinnett for comments.

Footnotes

auth_error_cmd =
  "notify-send -t 90000 \"pizauth error for $PIZAUTH_ACCOUNT\" \"$PIZAUTH_MSG\"";

auth_error_cmd =
  "notify-send -t 90000 \"pizauth error for $PIZAUTH_ACCOUNT\" \"$PIZAUTH_MSG\"";

Comments