What do we know about grant peer review in the health sciences?

Abstract

Background: Peer review decisions award an estimated >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members. Conclusions: We suggest funders should acknowledge, assess and analyse the uncertainty around peer review, even using reviewers’ uncertainty as an input to funding decisions. Funders could consider a lottery element in some parts of their funding allocation process, to reduce both burden and bias, and allow better evaluation of decision processes. Alternatively, the distribution of scores from different reviewers could be better utilised as a possible way to identify novel, innovative research. Above all, there is a need for open, transparent experimentation and evaluation of different ways to fund research. This also requires more openness across the wider scientific community to support such investigations, acknowledging the lack of evidence about the primacy of the current system and the impossibility of achieving perfection.

Criteria for assessing grant applications: A systematic review

“…Yet, pertinent literature reviews and compendia do not mention or only briefly discuss grant review criteria, as Moghissi, Love, & Straja noted in 2013. This observation still applies today and is evident in the most recent literature reviews on grant peer review (see Guthrie, Ghiga, & Wooding 2018;Guthrie et al 2019;Shepherd et al 2018). In this article, we therefore present a systematic review of studies on criteria for assessing grant applications.…”

Why can't we make research grant allocation systems more consistent? A personal opinion

“…Indeed, allocation decisions may be little or no better than random (Avin, ; Pier et al, ) even if peer review is accompanied by additional mechanisms such as review committees (Graves et al, ). It has thus been argued that we waste large sums of money in assembling and judging information related to our selection criteria, and that research funding systems would be far cheaper and no less effective, if based on a modified lottery rather than on peer review (Fang, Bowen, & Casadevall, ; Guthrie, Ghiga, & Wooding, ; MacKay, Kenna, Low, & Parker, ). As persuasive as the negative analyses of peer review are, our “best” researchers whose applications regularly appear in the upper percentiles are unlikely to support a completely random allocation.…”

“…One completely unintentional outcome is that our systems apparently tend to suppress innovation (Guthrie et al, )—the very thing that researchers, fund providers, and managers all agree is a crucial component of the development of knowledge. The agencies charged with distributing funds frequently express their desire for projects to break new ground ; their lexicon commonly includes innovation , significant advances, and transformative ideas .…”

“…Future consequences, if included at all, are not given a high weighting or are subsumed within broader “impact” criteria where they have little influence. Several studies have found that speculation, innovation, and impact are less likely to be rewarded by basic science funding schemes than the rigor of the scholarship (Abdoul et al, ; Boudreau et al, ; Braben, ; Guthrie et al, ). Proposals that are highly speculative (thus scoring low on a likelihood of success criterion) and submitted by relatively inexperienced researchers (scoring low in terms of track record ) may well be the ones needed to pave the way for future advances but tend to receive low scores from peer review (Boudreau, Guinan, Lakhani, & Riedl, ).…”

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

“…Reference is made in the first paragraph to a “recent systematic review” by Guthrie et al (2018 1 ) (and also in the third paragraph). We note that this publication doesn’t refer to itself as being a systematic review, and indeed, it is an update of a 2009 review which describes itself as a non-systematic review.…”