Go back

What goes on?

   

A fly-on-the-wall account of funding-panel deliberations

I’m a research development manager, but I’ve also served two terms as a public member of a research-funding panel, and I still review the odd proposal. I’m going to draw upon that experience to try to explain how funding panels tend to work. They obviously vary depending on funder and scheme. So treat this article as based on a true story, but with certain scenes invented for dramatic purposes.

Once upon a time, in a hotel conference room…

I am one of about 25 people sitting behind name plates and laptops around the outside of three sides of a horseshoe of tables. On the fourth side is a large screen. Most attendees are academic experts, drawn from a broad range of disciplines to cover the full scheme remit. 

There are a small number of non-academic reviewers. Also present are staff from the funder who will drive the PowerPoint, make sure the meeting runs smoothly, and remind panel members about the scheme rules, allowing the chair to focus on the business of the meeting. The chair has an important role, but more as a facilitator guiding discussions than a person likely to impose their will. 

We’ll be here all day, having already spent several days preparing. Everyone serving on a funding panel takes their role and responsibility seriously. It’s a lot of work to read and engage with every bid. In four years of doing this, I’ve never seen anyone underprepared.

Each panel member has been assigned a few applications to introduce to the panel. The lead introducer is usually the panel member whose expertise is closest to the proposal. Their role is to summarise the application in five minutes or less. If it’s a full application, this will include comments at outline stage, the applicants’ response, and from expert peer reviewers.

Expert peer-review comments are invaluable, and in practice we won’t fund anything that none of the reviewers support but most sets of reviews are mixed. I’d say that for around half of applications, reviewers’ comments contain at least one comment based on an obvious misunderstanding. The introducer will mention it and the panel will disregard it. Hostile or aggressive negative reviews are rare, and often disregarded.

After their summary, the lead introducer gives the panel their own opinion in terms of strengths and weaknesses, and score. Then two other introducers—including me or one of the other public representatives—add anything that’s been missed or overlooked. We then add our scores, though all introducers can change their scores after the discussion.

The introducers set the tone for the discussion. We don’t have long to discuss each application—perhaps 20 minutes in total for a full proposal. Discussions are civil and disagreements polite. 

Criticism-cascade

One phenomenon I have noted I call ‘criticism cascade’. Discussions start out positive, only for one critical comment to lead to another and another until the application’s fate is sealed. If that first criticism hadn’t been made, would the others have been prompted? I’ve often wondered.

After discussion, we submit our scores individually and electronically, and they appear (anonymised) on the PowerPoint. We have a standard scoring system but, more importantly, the panel has a shared understanding of what a six, a seven or an eight should be. When this is done, we move on to the next proposal.

At the end of the meeting we are shown an overall ranking, and where the funding cut-off is. The chair asks if we’re happy, especially if it is very close and/or the ranking does not seem to be reflective of the discussions. This is important because our role is not to assign grades to applications, but to rank them in priority order. We’re aware of the dangers of getting more critical as we go on, or being too positive or negative about the first application before we’re properly calibrated. Sometimes we’ll rescore.

There are more high-quality proposals than there is funding. Many excellent projects go unfunded and there’s nothing panel members can do about it. With outline applications, we’re only going to invite those that we think have a realistic chance of competing for funding—our chair’s usual tiebreaker question is “do we want to see this again?”

The moral of the story

What can we learn from this tale? In this case there are three main pointers for applicants. First, be confident that everyone is doing their best. Second, it pays to craft your proposal carefully to make it as easy as possible for panel members, and in particular for your introducers; it’s difficult for anyone to hold so much information about so many varied and complex proposals in their head at once. 

Third, apply or volunteer to serve on a funding panel—the experience of seeing decisions being made is invaluable. What’s more, it’s a vital service to your research community.

Adam Golberg is research development manager (charities) at the University of Nottingham

This is an extract from an article in Research Professional’s Funding Insight service. To subscribe contact sales@researchresearch.com