Study of Norwegian funder shows screening two-page proposals benefits both applicants and reviewers, says Marco Seeber
Writing and evaluating research proposals is very time-consuming and can generate considerable costs, both direct and indirect.
For example, a 2021 study estimated that putting together a proposal for the European Research Council takes between three and six months of one person’s effort. Factoring in a 10 per cent success rate, this means that each funded proposal amounts to between 2.5 and five years of work—roughly the length of a postdoc contract. And this leaves out proposals that aren’t submitted, as well as institutional and national investments to promote and support applications.
Some scientists have estimated that searching for funding takes up to 60 per cent of their time. For funders, the costs of evaluating proposals—administration and evaluation managers, reviewers and editors—have been estimated at around 20-35 per cent of the allocated budget.
Wasted time
Not surprisingly, funding agencies are experimenting with ways to reduce this effort. One solution adopted by some agencies worldwide is to replace the traditional, single-stage evaluation of a full proposal with a two-stage process, in which applicants would submit a short proposal for initial review, with a full application required only if this first stage is passed.
In many, perhaps most, research calls, distinguishing between bad and good proposals is relatively simple, while separating good from excellent ones is much more challenging. By speeding up the process of sorting proposals that are ‘possibly’ from those that are ‘definitely not’, two-stage evaluation allows effort to be focused on separating the ‘good’ from the ‘excellent’.
Introducing stage two
Norway’s largest funder of health research, the Dam Foundation, introduced two-stage evaluation in 2020. Prior to this, all its calls required a single 10-page proposal. This was replaced with an initial two-page proposal, with a 10-page follow-up from applicants that passed this stage.
To gauge the effect of the change, my colleagues and I compared the evaluations of 593 long proposals in the one-stage process with 668 short and 184 long proposals in the two-stage process. We also used results from a survey of applicants and reviewers on the amount of effort they put into each.
Applicants in the one-stage process estimated that writing a long proposal took an average of 37 person-days. In the two-stage process, writing short proposals consumed an average of 16.75 person-days and 36 person-days for long proposals. Since nearly three-quarters of proposals were rejected in the first stage, the two-stage process saved on average 38 per cent of applicants’ time.
These are huge savings. To put them into perspective: a decade ago, a study of the Australian National Health and Medical Research Council estimated that researchers spent 550 working years in preparing 3,727 proposals—equivalent to AU$66 million (£35m) in salary costs. In a similar setting, a two-stage process would save roughly 209 working years and AU$25 million.
According to the reviewers’ estimates, the two-stage procedure also reduced the time they spent evaluating proposals by an average of 28 per cent. A study of a switch to two-stage evaluation at the UK’s National Institute for Health and Care Research found a very similar result.
Applicants welcomed the new procedure, with only 2 to 4 per cent of those surveyed reporting dissatisfaction. The number of applications did not change.
Two-stage evaluation was also more consistent: on average, reviewers disagreed less in their assessments of short proposals than of long proposals. This might be because evaluators assessing short proposals avoid extreme scores, or that reviewing long proposals leads to cognitive fatigue and more erratic judgment.
Recognising trade-offs
The greater reliability may also be a result of cutting the number of evaluation criteria from nine to four: it’s been suggested that having many evaluation criteria also reduces accuracy. Funders need to understand these trade-offs between detail, effort and accuracy in peer review.
Two-stage evaluation might bring other changes that are harder to detect. For example, some types of proposal and applicant may be comparatively better off in a short rather than in a long version, or vice versa. Our analysis provided some hints in this regard: we found that the scores of short proposals that passed the first stage remained high, but also that their rank changed.
Our findings add to evidence that a two-stage process can make evaluation of research proposals more efficient and ease the burden on the research community. Experiences like that of the Dam Foundation provide a template of how it can be implemented.
Marco Seeber is a professor in the Department of Political Science and Management at the University of Agder, Norway