Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices

Abstract

BackgroundPeer review of grant applications has been criticized as lacking reliability. Studies showing poor agreement among reviewers supported this possibility but usually focused on reviewers’ scores and failed to investigate reasons for disagreement. Here, our goal was to determine how reviewers rate applications, by investigating reviewer practices and grant assessment criteria.Methods and FindingsWe first collected and analyzed a convenience sample of French and international calls for proposals and assessment guidelines, from which we created an overall typology of assessment criteria comprising nine domains relevance to the call for proposals, usefulness, originality, innovativeness, methodology, feasibility, funding, ethical aspects, and writing of the grant application. We then performed a qualitative study of reviewer practices, particularly regarding the use of assessment criteria, among reviewers of the French Academic Hospital Research Grant Agencies (Programmes Hospitaliers de Recherche Clinique, PHRCs). Semi-structured interviews and observation sessions were conducted. Both the time spent assessing each grant application and the assessment methods varied across reviewers. The assessment criteria recommended by the PHRCs were listed by all reviewers as frequently evaluated and useful. However, use of the PHRC criteria was subjective and varied across reviewers. Some reviewers gave the same weight to each assessment criterion, whereas others considered originality to be the most important criterion (12/34), followed by methodology (10/34) and feasibility (4/34). Conceivably, this variability might adversely affect the reliability of the review process, and studies evaluating this hypothesis would be of interest.ConclusionsVariability across reviewers may result in mistrust among grant applicants about the review process. Consequently, ensuring transparency is of the utmost importance. Consistency in the review process could also be improved by providing common definitions for each assessment criterion and uniform requirements for grant application submissions. Further research is needed to assess the feasibility and acceptability of these measures.

What do we know about grant peer review in the health sciences? An updated review of the literature and six case studies

“…They found that this affected the outcome for 29 per cent of the proposals considered, also noting that the degree of reliability varied greatly between panels, suggesting that in some disciplines review panels find it more difficult than in others to agree on a proposal's quality. Some of this variability might be explained by differences in reviewer behaviour, as observed by Abdoul et al (2012), who found that the time spent assessing each application, assessment methods, and relative weighting of different criteria in the overall assessment of proposals differed between reviewers.…”

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

“…The study showed that even this minimal training increased the reliability of grant rating for both novice and experienced reviewers. Abdoul et al 14 underscored the importance of reviewer training in grant assessment criteria.…”

The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability

“…Higher IRR in peer review panels has been reported to occur when the evaluation priorities of the funding agency are in line with those of the reviewers (Abdoul et al, 2012). However, for criteria where this priority is yet to be determined, and is in constant debate, this alignment of views is unlikely to happen naturally and some level of reviewer education is deemed necessary to increase IRR and also to maintain the legitimacy and validity of the review process itself.…”