One Monday a few weeks back, I found myself unable to send emails via my work account. I had received no advanced warning and there was no clue in the error message I was seeing as to what the cause might be. I was briefly tempted to view my cut-off from work as a good thing before, irritatingly, the sense of duty my parents instilled in me kicked in. After a while I realised that the problem was that my msmtp setup was using basic authorisation – what the rest of us think of as a traditional username and password – to send emails. Not only does my work’s Microsoft Exchange server no longer accept basic authentication, but Microsoft are killing it off for everyone before the end of the year — more people are thus about to experience the same confusion I experienced.
A common alternative to basic authorisation is OAuth2. From the perspective of this post [1], OAuth2 requires you to use a web browser to authenticate yourself (which might then involve two-factor authentication e.g. via your mobile phone) after which you can obtain time-limited access tokens. You can use access tokens to authenticate yourself to, for example, an IMAP or SMTP server. As irritating as I sometimes find two-factor authentication [2], there is no doubt that OAuth2 is more secure overall than basic authentication.
In my case, I solved my immediate problem with email-oauth2-proxy, which has a very cunning approach to the problem (it’s a semi-transparent proxy that provides the illusion of basic authentication to user clients while performing OAuth authentication to users) but it occasionally stalled and required me monitoring an xterm for authentication links which doesn’t sit neatly in my email setup. I then tried mailctl but it segfaulted on me (perhaps because of an incompatibility with OpenBSD) and I’m not familiar enough with Haskell to debug such things.
A few days later I thus found myself writing a “traditional Unix daemon” for obtaining OAuth2 tokens. I ended up with a few requirements (some of these will only make sense at this point in the post if you know OAuth2; if you don’t, I hope these will make sense by the end of the post).
- Asynchronous notification of authentication requests. I do not want to monitor an xterm, or a log file, for authentication URLs.
- Background refreshing. I would like the illusion of immediately available, valid, access tokens whenever possible.
- No secrets stored on disk. Other OAuth2 token requesters do this, and then have to jump through hoops to protect the secrets. Keeping secrets in RAM reduces such problems substantially.
The result of my labours, pizauth, has just had its first alpha 0.1.0 release. For me, “alpha” means “this at least roughly works, but there might be obvious bugs or oversights, and while I hope the user interface won’t change, I will modify it if flaws become apparent.” Probably the main set of people who will be interested in pizauth are those who use traditional Unix email tools (e.g. extsmail, fdm, neomutt, isync, or msmtp) to read or send email. If you are such a person – or even if you’re not! – I would welcome testing on pizauth: does it work with your email providers? with your local software? is the documentation complete and understandable? do you encounter bugs? etc. Here’s what my particular configuration of pizauth looks like when I use neomutt to read an IMAP folder:
Basic setup
If, like me, you have no previous experience of OAuth2, it can be
surprisingly hard to envision what it actually entails (see e.g. the OAuth2 site for pointers). In pizauth’s case, we
first create a config file at ~/.config/pizauth.conf
. For example,
let’s assume I want to be able to read and send my work email via Exchange’s
IMAP and SMTP servers:
account "work" { auth_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize"; token_uri = "https://login.microsoftonline.com/common/oauth2/v2.0/token"; client_id = "<your-client-id>"; client_secret = "<your-client-secret>"; scopes = [ "https://outlook.office365.com/IMAP.AccessAsUser.All", "https://outlook.office365.com/SMTP.Send", "offline_access" ]; login_hint = "<your-email-address>"; }
In essence, OAuth2 requires a URI to authenticate against
(auth_uri
), a URI to obtain tokens from (token_uri
),
and the “scope” (perhaps more easily thought of as “permissions”) you want
access to (scopes
). Although it’s not mandatory, it helps to add
login_hint
which is your email address.
client_id
and client_secret
are, in my opinion, blots on
OAuth2: they’re best thought of as being a fixed username and password. For unmonitored
server applications, having a password makes sense but for end
user applications it is simply security by obscurity. If you’re lucky, your
OAuth2 service provider will only require client_id
and will tell
you what a valid value for it is. If you’re moderately lucky, your OAuth2
service provider will provide a mechanism to let you generate unique values for
both client_id
and client_secret
. If you’re unlucky,
you’ll have to guess what valid values your OAuth2 service provider will accept
— often other people on the internet will have sussed out values that
will work. I hope that, in the future, user-facing systems will stop thinking
that client_id
and client_secret
are anything other
than a needless barrier to OAuth2 usage.
With configuration complete, we then start pizauth’s server:
$ pizauth server
At this point, absolutely nothing visible will happen: pizauth will daemonise itself, and wait for you to ask it to do something. What you probably want to do is to show the current OAuth2 access token:
$ pizauth show work ERROR - Token unavailable until authorised with URL https://login.microsoftonline.com/common/oauth2/v2.0/authorize?access_type=offline&code_challenge=xpVa0mDzvR1Ozw5_cWN43DsO-k5_blQNHIzynyPfD3c&code_challenge_method=S256&scope=https%3A%2F%2Foutlook.office365.com%2FIMAP.AccessAsUser.All+https%3A%2F%2Foutlook.office365.com%2FSMTP.Send+offline_access&client_id=<your-client-id>&redirect_uri=http%3A%2F%2Flocalhost%3A14204%2F&response_type=code&state=%25E6%25A0%25EF%2503h6%25BCK&client_secret=<your-client-secret>&login_hint=<your-email-address>
Perhaps unsurprisingly there isn’t a token to show yet. Instead what we’ve got
is a (half kilobyte long) URL. If we paste that URL into a web browser,
Microsoft will ask us to authenticate ourselves
in the normal fashion for your organisation, which might involve two-factor authentication and so on (if
you’re already authenticated in that browser, and lucky, you probably won’t have
to do anything). When authentication in the browser is complete, you will be sent to a
webpage which says simply “pizauth processing authentication: you can safely
close this page.” By the time you’ve closed that window, the chances are that pizauth
will have completed its background authentication work. With luck, pizauth show
will then show an access token [3]:
$ pizauth show officesmtp DIASSPt7jlcBPTWUUCtXMWtj9TlPC6U3P3aV6C9NYrQyrhZ9L2LhyJKgl5MP7YV4
That access token can then be used to read and send email. So now you need to configure your email software to call pizauth. For example, I configure msmtp as follows:
account work auth xoauth2 tls on syslog on passwordeval pizauth show work user your-email-address port 587 host smtp.office365.com
For neomutt and IMAP [4]:
set imap_user = "your-email-address" set folder = "imaps://outlook.office365.com:993" set spoolfile = "+INBOX" set ssl_force_tls = yes set imap_authenticators="oauthbearer:xoauth2" set imap_oauth_refresh_command="pizauth show work"
Asynchronous notifications
It can be rather irritating to manually call pizauth show
to find out
the URL you need to put into your browser, especially as periodically (ranging,
typically, from days to months) you will find you need to reauthenticate
yourself against the OAuth2 server. Put another way, authentication requests
are potentially asynchronous.
pizauth thus allows you to run arbitrary shell commands when authentication
is needed via the global auth_notify_cmd
setting in
pizauth.conf
. This could simply open the authentication URL up in
your default browser:
auth_notify_cmd = "open \"$PIZAUTH_URL\"";
When pizauth runs the shell command, it sets two environment variables:
$PIZAUTH_ACCOUNT
is the account name (in this case “work”); and
$PIZAUTH_URL
is the authentication URL I need to run in my
browser.
Personally, I want a pop-up to display on my XFCE desktop via
notify-send
. Most notification daemons can parse basic HTML, which
is great, because I can then click on a link in the notification to go to a
browser and authenticate myself. However, XFCE’s notification daemon doesn’t
parse ‘&’ characters correctly, so I have to transform them into
‘&’ with sed
. That means that I’ve ended up with a
slightly scary looking command:
auth_notify_cmd = "notify-send -t 30000 'pizauth authorisation' \"<a href=\\\"`echo $PIZAUTH_URL | sed 's/&/&/g'`\\\">$PIZAUTH_ACCOUNT</a>\"";
The notification stays alive for 30 seconds (-t
takes its
argument in milliseconds) [5]. If I miss that
notification, auth_notify_cmd
will be rerun every 15 minutes (the
default value for the auth_notify_interval
setting).
When and where to start pizauth
pizauth is a traditional background Unix daemon, so you can start it whenever
you want and it will happily put itself into the background and stay there.
However, where is a good place to call pizauth server
from?
Personally I do so in my
~/.xsession
file so that pizauth starts when I log into X: that
means that it picks up the necessary environment variables for
notify-send
to work.
However, there’s no reason that you can’t start pizauth’s server elsewhere e.g. in your shell’s login file. The server checks for the existence of a (functioning) pizauth server and won’t start if one exists, so you don’t have to worry about accidentally starting two pizauth servers.
Background refreshing
OAuth2 access tokens typically have a short expiry time (around 1 hour is common). Although largely hidden from users, there are also refresh tokens which allow one to obtain new access tokens (a reasonable intuition is that refresh tokens are a sort of “superuser token”). Refresh tokens typically have a long expiry time (days, weeks, or months). One can use a refresh token to obtain a new access token at any point (even before the access token has expired), but some systems put a limit on how many new access tokens you can obtain in a period of time.
pizauth thus faces an interesting challenge: when should it refresh
access tokens? It could wait until the first show
request after an
access token has expired and then refresh it, but that can cause
irritating pauses. It could refresh the access token upon every
show
request but as well as irritating pauses this could lead to
users exceeding the access token limit. Both these approaches also imply that
displaying access tokens should block until refreshing has finished.
I suspect that blocking in this way will cause users to assume
that displaying access tokens always succeeds. That assumption breaks down at
the point that refresh tokens expire and reauthentication is required.
The approach I’ve taken is two fold. First, pizauth show
is
non-blocking. As we saw above, it will immediately return with an error when it
is first called; it will also return with an error if it has been unable to
refresh an expired access token. Second, pizauth has a background thread which
tries to refresh access tokens just before they’re about to expire (by default 90 seconds
before, controlled by the per-account refresh_before_expiry
setting). In general this means that pizauth show
shows access tokens
immediately.
However, access tokens can be externally revoked [6],
so there’s no guarantee that the access token pizauth displays is valid.
pizauth provides two mechanisms to resolve this problem. Users (or programs)
that realise an access token is no longer valid can explicitly call
pizauth refresh
to forcibly refresh an access token: if
refreshing fails, this returns an error (with an authentication URL). However,
in general, we don’t want users to have to explicitly request refreshing. pizauth
thus provides a “backstop” setting refresh_at_least
setting (which
defaults to 90 minutes) which forcibly refreshes an access token even if hasn’t
expired. In practise this means that, at worst, pizauth will show invalid access
tokens for a fixed period of time before either obtaining a new access token or
notifying the user that they need to reauthenticate.
In summary, pizauth’s default settings mean that users don’t really have to worry about refreshing: it should mostly “just work”.
Config reloading
Many Unix daemons have a way of reloading their config file without having to
fully shutdown and restart (often via the dicey SIGHUP
signal).
Since pizauth does not store any state on disk, shutting down and restarting is
particularly irritating, as it would require reauthenticating every account.
pizauth thus has a pizauth reload
command (which doesn’t use
SIGHUP!
) which causes pizauth to reload its configuration.
In my experience, configuration reloading is often done in a crude fashion.
I soon realised that wasn’t acceptable for pizauth where sloppy configuration
reloading could cause access tokens being handed out to the wrong server (e.g.
if you swap two accounts’ auth_uri
s; or delete an old account and
create a new one with the same name but otherwise different details). It might
not be obvious, but this is a nightmare scenario: if I give an access token
intended for server S1 to server S2, then S2 could
use that access token to illicitly obtain services from S1. This isn’t
very likely to happen, but I wanted pizauth to make sure that users could never
shoot themselves in the foot.
When pizauth reloads the user’s configuration, any account which is changed in any way has its tokens removed (i.e. the user will have to reauthenticate from scratch), thus removing the shotgun entirely. Because this is an important security property, pizauth has a carefully crafted internal API which causes it to crash if this property could be violated. I’ll probably write a more detailed post on that at a later point.
Summary
pizauth solves a problem I have, and which I know some other people have too. My aim has been that it is, as far as possible, a “configure once and then leave alone for ever” daemon. If you’re interested in such a program, please try out the alpha release and let me know what problems you find (I am bound to have made mistakes and overlooked things)! For example, I and my set of initial testers have only tested pizauth on Microsoft and Gmail’s OAuth2 servers: it might be the case that other systems need slight tweaks to get them to work. But, hopefully, you will find pizauth a useful tool!
Acknowledgements: thanks to Edd Barrett, Dejice Jacob, and J. Ryan Stinnett for comments.
Footnotes
OAuth can also be used for things like servers authenticating amongst themselves. I’ve not really thought about that aspect a great deal, so I don’t pretend to understand the details or consequences of this.
OAuth can also be used for things like servers authenticating amongst themselves. I’ve not really thought about that aspect a great deal, so I don’t pretend to understand the details or consequences of this.
It does not help that I live in a house whose stone walls appear to have been designed to be just the right thickness to block mobile phone signals.
It does not help that I live in a house whose stone walls appear to have been designed to be just the right thickness to block mobile phone signals.
I could safely show you a real, expired, access token, but they’re rather long: Exchange’s are ludicrously so (2044 bytes), Gmail’s less so (211 bytes). I’ve thus put in some shorter random data.
I could safely show you a real, expired, access token, but they’re rather long: Exchange’s are ludicrously so (2044 bytes), Gmail’s less so (211 bytes). I’ve thus put in some shorter random data.
set imap_authenticators="oauthbearer:xoauth2"
tries using the more
standard “oauthbearer” mechanism first, before falling back to
xoauth2
. If you know your provider only uses one mechanism, you
can delete the other.
set imap_authenticators="oauthbearer:xoauth2"
tries using the more
standard “oauthbearer” mechanism first, before falling back to
xoauth2
. If you know your provider only uses one mechanism, you
can delete the other.
Unsurprisingly, there’s also auth_error_cmd
which will be run when
an error occurs with a message in $PIZAUTH_MSG
. My setting is:
auth_error_cmd = "notify-send -t 90000 \"pizauth error for $PIZAUTH_ACCOUNT\" \"$PIZAUTH_MSG\"";
Unsurprisingly, there’s also auth_error_cmd
which will be run when
an error occurs with a message in $PIZAUTH_MSG
. My setting is:
auth_error_cmd = "notify-send -t 90000 \"pizauth error for $PIZAUTH_ACCOUNT\" \"$PIZAUTH_MSG\"";
To my surprise, OAuth2 doesn’t require systems to expose a way for users to revoke access or refresh tokens. This seems an oversight to me.
To my surprise, OAuth2 doesn’t require systems to expose a way for users to revoke access or refresh tokens. This seems an oversight to me.