What Makes a Good Research Proposal?

Blog archive

Research is rarely quick: it takes time, people, and (in most cases) equipment to complete. To convince those in control of the purse strings that we should be given the necessary resources to tackle a research problem, we create a research proposal. This might be: a written document, a presentation, or a series of conversations; presented to academia or industry; and aimed at convincing a manager or an external funding agency (or even oneself!).

Whatever the situation, I believe that good proposals share much in common. Unfortunately, many guides I’ve seen to creating proposals focus, sometimes very cynically, on the quirks or biases, real and imagined, of funders. In contrast, in this post I’m going to try to focus on the ideal core of a proposal, because that has inherent value in its own right, and also because it can be easily adapted to the specifics of a given funder.

By focussing on this core, I’m hoping that I might help you answer two questions: “do I have an idea that’s worth asking for resources to tackle?” and “how do I present that idea in a way that other people will best understand?” To make my life easy, I’m going to frame this discussion in terms of “readers” and “writers”, though I hope the ideas generalise beyond written proposals, and “funders” as a shorthand for those in control of resources.

The three fundamental questions

To my mind a good proposal must address three fundamental questions:

1. What is the problem being tackled?

2. Why is the problem worth being tackled?

3. What is the insight that makes tackling the problem plausible?

Although the third of these is by far the most important, the first two set up the necessary context to understand the third.

What is the problem being tackled?

The aim here is to describe a deficiency in our current knowledge (see my previous post on where one might find worthwhile research problems). This is harder than it first seems: every reader must feel they’ve at least roughly understood the problem, otherwise they won’t understand any other part of the proposal, but experts must not feel the problem has been oversimplified. Put another way, at least in the sort of science / technical proposals I’m familiar with, some technical jargon is inevitable, but I try to keep it to a minimum.

For example, I might say “The standard implementations of programming languages such as Python and Ruby run 5x slower than state-of-the-art alternatives with Just-In-Time (JIT) compilers, but many programs are only compatible with the standard implementations, which thus remain dominant.” Even if you don’t know the particular programming languages mentioned, let alone what a “JIT compiler” is, you’ll probably understand that a trade-off space of “compatible but slow or incompatible but fast” isn’t ideal, and that a 5x difference in performance between the two is a clear deficiency.

Why is this problem worth being tackled?

There are always more problems than available resources: a good proposal needs to motivate why the problem it describes is worth prioritising. In the situations that I deal in, a brief motivation is sufficient, though in some other contexts (particularly those where you’re asking for significant resources) you might need to give more expansive, quantitative motivation. With that bias in mind, and continuing our running example, I might say that “Slow programming languages waste programmer time, give users a poor experience, require unnecessary quantities of servers, and waste energy.”

Unfortunately, most proposals do not provide an adequate motivation. The most common case is that proposals are put together by experts who either forget that other people need to be provided with a motivation or who think that the motivation is too obvious to need stating.

Another case is when the proposers lack experience of the wider context in which their research fits. This is common in proposals from those who’ve recently finished a PhD, whose stated motivation often boils down to “I’m going to do more of the research I did in my PhD” which isn’t a motivation at all. A simple hint or two from an old hand that their research needs to be placed in a wider context is often enough to solve this.

At the opposite end of the career ladder, sometimes those who’ve been working in a certain area for a long time can no longer imagine working on any problems other than those they’ve worked on before. The motivation they give may not be exactly incorrect, but somehow stale, and unconvincing. I’m not sure there’s a general solution for those stuck in this particular rut, though moving to a different research area reinvigorates some people.

Finally, and there isn’t a nice way of saying this, sometimes a good motivation for a problem doesn’t exist. Sometimes people want to tackle problems that have already been solved (!), or whose benefits are so tiny as to render them uninteresting. Fortunately, such cases are rare.

What is the insight that makes tackling the problem plausible?

The most important part of a good proposal is an insight that explains how the proposers might plausibly make progress when tackling the problem [1]. I use the word “plausibly” deliberately: there can be no guarantee that the proposers will definitely succeed, or even make progress. Also, despite the common emphasis on “novelty”, the insight might not necessarily be novel in the sense that no-one has had the insight before, but one should at least make clear why no-one has taken the insight sufficiently far.

Despite the low bar that “plausibly” implies, I have read an astonishing number of proposals that, caricatured only slightly, say “Gravity causes people to break bones. If we can stop gravity, people will stop breaking bones. We will research gravity.” The first two sentences are inarguably true, but the third sentence provides no insight at all, and makes clear that the entire proposal is flawed. Richard Hamming made a very similar point [2]:

The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack.

What makes a good insight? I’ll tackle the common case, which is where we are not the first people to try tackling a worthwhile problem: we need to explain why we stand a chance of succeeding where our predecessors did not (or, looked at in the opposite way, why we’re not going to fail in the same way that our predecessors did). There are three main ways that we might succeed where they did not: mass; a change in context; or a novel idea.

Sometimes the reason that our predecessors failed is that they simply did not have sufficient resources to tackle the problem at hand. If for example, you want to validate the existence of the Higgs boson, you’re not going to get very far with a circular pipe, a torch, a few magnets, and a domestic electricity supply, even though those are the basic ingredients of Cern’s Large Hadron Collider (which cost around £6bn). In such cases, simply throwing a greater quantity of resources at the problem is the necessary condition for plausibly solving it.

In other cases, our predecessors had the misfortune for their work to depend on an inadequate supporting pillar [3]: if that pillar has now been strengthened, simply tackling the original problem again may lead to success. For example, a lot of 1980s software research either failed or underwhelmed because computers of the day were too slow: repeating that research on faster computers can often lead to success [4].

Finally, sometimes we have a novel idea that promises an entirely new route towards tackling a problem. That novel idea can be as simple as putting two things together that no-one has previously thought of putting together [5]. Continuing our running example, I might say that “Meta-tracing is a technique which can automatically speed up specially written programming language implementations. We will adapt it so that it can automatically speed up existing language implementations which will still maintain full compatibility with their unaccelerated forebears.” I wouldn’t expect many readers to know what meta-tracing is (it’s my novel idea in this context), but I can explain it in more detail later. However, I do hope that every reader gets the sense that my insight is that we can generalise something more than has previously been thought possible.

Secondary questions

A good proposal will address several other questions that, though of less importance than the fundamental questions, still carry useful information.

Are there any additional consequences if the main problem is solved? This question follows directly on from the three fundamental questions and allows the proposers to show they understand the wider context in which their work sits. By extension, this also provides further motivation for why the problem is worth tackling in the first place. Continuing our running example I might say “Making it cheaper to make faster programming languages will allow more people to experiment with new programming language designs” or “Improving programming language performance will decrease the environmental impact of servers.”

Why are the project proposers the right people to carry out the research? There seem to me to be two major cases. In an established field, credentials are important: if you’re proposing solving a mathematical problem that’s stumped people since 1670, you should be able to demonstrate thorough expertise in mathematics. In contrast, if you’re proposing working on a fairly new problem where by definition there can be few experts, then the credential bar is a lot lower, though it never quite goes to zero! My experience is that it’s also important to understand funder expectations: academic funders tend to place significant weight on experience and credentials; start-up funders, in contrast, tend to look for energy and optimism.

What is the plan of work for tackling the problem? In my opinion, a plan of work for a research project should reflect the aphorism that “plans are worthless, but planning is everything”. The aim isn’t to sets out work in minute detail, but to show that there is at least one high-level series of steps that might plausibly lead to solving the problem. I tend to set out the initial steps in a moderate level of detail, with the later steps being much briefer, since the further we look ahead the less certainty we can have. The plan of work has four advantages which I think are easily forgotten. First, it is an excellent sanity check as to the feasibility of the research problem and insight. Sometimes when I start planning I realise that I will have to first solve other problems that require an entire proposal in their own right! Second, it gives a sense of the required scale of research. Third, it gives a concrete initial direction that allows those eventually working on the project to get going quickly. Fourth, it gives people a context in which to make timely decisions about the inevitable surprises they will encounter when undertaking the research.

How many resources are needed? We have to put hard figures on the resources we’re asking for, the chief components of which tend to be people and equipment. Surprisingly, it seems equally common for people to request too few as too many resources: the former will cause the research to fail; the latter leads to waste. In either case, the most frequent cause seems to be that people guess the maximum level of resources they think the funder will give them and then ask for that, irrespective of the research being proposed. It is much better, in my opinion, to use the plan of work to make a good estimate of the resources required and then work out who might fund that. If this exercise shows that more resources are required than can possibly be allocated, then either the problem being tackled needs to be cut down, or another source of resources needs to be sought.

What is the level of risk? Many proposals have a tendency to imply that their research is guaranteed to succeed. In general, if research is guaranteed to succeed then it’s not really research! Good research has an element of risk that matches the potential reward: low-reward problems can be low risk; but high-reward problems can be high risk. I tend to highlight what I think are the most risky parts of the research and explain either what a fallback option is or why there can be no fallback. For example, in our running example, I might say “We will use the existing widely used compiler framework XYZ for code generation. However, since it has not previously been used in the context of tracing, we cannot guarantee that it will produce fast code for us. If that risk manifests, we will write our own code generator: this would be a reasonably significant effort and would cause us to divert some resources from other planned work.” It can feel risky to hand reviewers a stick with which to beat you but good reviewers, who I prefer to optimise for, will punish you for the absence of the stick!

Style

Research proposals are very different to anything else a researcher does: they’re necessarily speculative and full of unjustifiable (though hopefully not outrageous!) claims. Without a careful, succinct explanation, it’s easy for those reading or listening to a proposal to assume things that were never intended.

The easiest way to keep readers – and, indeed, writers! – on track is to set out the context at the very beginning of the proposal. Over time I have become insistent that proposals start with summary answers to the three main questions from earlier, allocating perhaps 2 to 3 sentences to each question. This degree of terseness does not come naturally to most of us and some people feel that it trivialises their research. Experience has taught me, however, that if I can’t summarise my ideas in this way, then either I’m not yet ready to write the proposal, or I don’t have a good idea for a proposal.

It might sound odd to suggest that too much detail is a bad thing but there’s a limit to how much a proposal can sensibly say about the future. Sometimes proposals try filling the gap by making vague or outlandish claims, with obvious consequences. More commonly, proposals go into vast detail about related work, hoping that helps readers better understand the future. Unfortunately, all it does is create a demand in the reader’s mind for even more detail about related work, which is a death spiral from which no proposal can recover. When I’m reviewing a proposal, I do need to be convinced that the proposers understand the state-of-the-art sufficiently well, but often a single sensible reference to a single recent paper in each area of interest is all that I need to be convinced.

Something that I’ve alluded to multiple times above is the tension between technical accuracy and readability. I’ve lost track of the number of proposals I’ve read that use so much jargon that they’re unintelligible by anyone outside a tiny cabal. Although less common, some proposals so dumb down their description that they end up unintelligible in a very different way. There can be no hard and fast rules about what the right balance is, other than that the two extremes are bad. Whenever I’m in doubt, I introduce each new point with a brief, accessible, high-level description that I hope all readers can roughly understand, followed by a slightly lengthier and more technical description for experts. Sometimes this can even work in a single sentence! Ultimately I tend to be heavily informed by readers of drafts: if they complain that something doesn’t make sense, I assume the fault is mine, not theirs.

Finally, the best proposals I’ve read convey a sense of excitement. This excitement cannot, and should not, be conveyed by outlandish claims — indeed, such claims diminish both the proposal and the proposers. Rather, when a proposal tackles a meaningful problem and has an excellent insight, I as the reader cannot help but be excited by what will happen when the research is carried out. I then really want the proposal to be allocated resources and I’m prepared to go out of my way to try and make that happen. Conversely, as a proposer writer, if I don’t feel that excitement myself then it’s unlikely that anyone else will either.

Adapting to the funder

In most of the above, I have presented the Platonic ideal of proposals. In reality, we always have to compromise on this ideal to reflect the quirks and biases of funders. These tend to accumulate in a funder for two main reasons. First, they’re hoping it encourages higher quality, or at least more consistent, proposals. Second, they want to extract specific information that proposers would otherwise not include. Broadly speaking, the more quirks and biases a funder requires, the harder it is to write a good proposal, and the more that inside knowledge (or, at least, experience) confers a meaningful advantage.

A common quirk is where funders want proposals split into predefined headings. Frequently there is duplication between the headings, forcing the proposers to find two ways of saying essentially the same thing. Sometimes the headings simply don’t make much sense, at which point I do my best, and hope that (as is commonly the case) the reviewers will also be baffled by such headings. Funders also often ask for sections in a specific order which rarely reflects the best way to explain the proposal. A particular favourite of mine is a funding source that asks for project objectives before the summary. Inexperienced proposers torture themselves trying to explain research objectives to a reader who doesn’t yet know what the research is. Experienced proposers simply include a quick summary in the objectives.

Biases come in several flavours, but one example hopefully suffices. It is increasingly the case that funders ask proposers to explain how their research meets wider societal goals. Since these goals are always in flux, funders typically have a hard job articulating clearly what goals they actually care about. In general, the best proposals I’ve seen prioritise maintaining the integrity of their core ideas while briefly explaining how those ideas map onto the required societal goals.

There is a final kind of compromise that is not exactly a quirk or bias: where the funder requires information that they hope nudges the proposers in the right direction, such as diagrammatic representations of work plans, plans for exploitation, and the like. In general, these either duplicate information already present in the proposal, or are padding relative to the core ideas: either way they don’t tend to impact the core ideas as much as the other quirks and biases I’ve listed.

For those about to review proposals

Most of what I’ve written above is aimed at those putting together a proposal, but I also have some thoughts for those who solicit or review proposals.

First, proposals should be short, to avoid wasting the time of both proposers and reviewers. For written proposals, I think the optimum size is about 2 or 3 sides of A4: enough to get across the core ideas while not giving the proposers enough rope to hang themselves. I have less experience with oral proposals, but I would aim to keep them equivalently short.

Second, choosing who evaluates proposals is critical: one needs people who are open-minded enough to recognise good ideas without being indiscriminate dreamers. This is a rarer combination than I once expected. Some people seem to enjoy finding elements of the unknown in a proposal and using those as a reason to reject it — as if elements of the unknown should not exist in a proposal! In contrast, some other people seem unable to critically evaluate ideas, and end up being willing to allocate resources to charlatans. Unfortunately, I don’t know how one can pick good evaluators without first seeing them in action.

Third, the more requirements one makes of proposers, the worse the resulting proposals will be. Perhaps more accurately, it tends to particularly lower the quality of what would otherwise be the best proposals. In other words, the more requirements one imposes, the harder it is to extract the core ideas that allow one to differentiate good from bad proposals.

Summary

I’ve reviewed a lot of research proposals, and written a few myself, and I think it’s fair to say that writing good research proposals is a skill which does not come naturally to most of us. The first one I ever wrote, for example, was truly awful [6] and there have been a few subsequent stinkers. I think, though, that it is a skill that can be learnt.

Although it is always tempting to bemoan the need for proposals – especially because human evaluators have a significant degree of randomness – proposals have value beyond just convincing funders. Writing a proposal forces me to think deeply about the research I want to do, and how I want to go about it: I often end up realising there are both new challenges and opportunities.

Oh, and the examples I used above were taken from one of my recent proposals — if you thought they were rubbish, you might want to disregard everything I’ve said!

Acknowledgements: thanks to Tyler Cowen and Stephen Kell for comments.

Footnotes

Comments