Evaluation of stakeholder views on peer review of NIHR applications for funding: a qualitative study

Abstract

ObjectivesInnovations resulting from research have both national and global impact, so selecting the most promising research studies to fund is crucial. Peer review of research funding applications is part of the selection process, and requires considerable resources. This study aimed to elicit stakeholder opinions about which factors contribute to and influence effective peer review of funding applications to the UK National Institute for Health Research (NIHR), and to identify possible minor improvements to current processes and any major changes or potential innovations to achieve a more efficient peer review process.DesignQualitative interviews with 30 stakeholders involved in the peer review process.ParticipantsParticipants were drawn from three NIHR coordinating centres and represented four types of stakeholders: board members with responsibility for making funding decisions, applicants, external peer reviewers and NIHR staff.MethodsAll interviews were conducted by telephone apart from three that were face to face with NIHR staff. Data were analysed using a thematic template method.ResultsThe responses from NIHR staff, board members and reviewers differed from those received from applicants. The first three groups focused on how well the process of peer review did or did not function. The applicants mentioned these points but in addition often reflected on how their personal application was assessed. Process improvements suggested included: developing a more proportionate review process; providing greater guidance, feedback, training, acknowledgement or incentives for peer reviewers; reducing the time commitment and amount of paperwork; and asking reviewers to comment on the importance, strengths and weaknesses of applications and flaws which are potentially ‘fixable’.ConclusionsOverall, participants were supportive of the need for peer review in evaluating applications for research funding. This study revealed which parts of the process are working well and are valued, and barriers, difficulties and potential areas for improvement and development.

The CTSA External Reviewer Exchange Consortium (CEREC): Engagement and efficacy

“…A possible future contribution of CEREC is to examine alternative approaches to selecting proposals for funding. Considerable hub resources are devoted to the review process across the entire CTSA national network; a fact that reflects the generally held confidence in peer review as a method of evaluating research funding proposals [5,6]. A small but growing literature has begun to investigate the role and dynamics of peer review in the funding decision process, and there has been a call for more systematic research to identify optimal processes for selecting science for funding that is both innovative and likely to be successfully executed [14,15].…”

“…The peer review process has long been used to provide "a system of institutionalized vigilance" [1] in the self-regulation of scientific communities. Although there is vigorous debate about how impartial the peer review process is [2], how reliable it is [3], and whether peer review scores are predictive of research productivity [4], the utility of peer review in the evaluation of funding proposals has been repeatedly affirmed [5,6] and the scientific community relies heavily on this process to evaluate competing proposals for funding. One approach to reducing potential sources of bias in peer review is to minimize the degree of affiliation between the applicant and the reviewer and maximize the level of expertise of the reviewer relative to the topic of the proposal [2].…”

What makes an effective grants peer reviewer? An exploratory study of the necessary skills

“…In one of the first person reviewer narratives, Member [23] emphasizes the importance not only of preparation, but also of knowing how to prepare. Turner et al [20] focused on process…”

“…Mow [19] describes the characteristics panel members use to assess proposals, suggesting elements of panel review elements that funding agencies may benefit from, including characteristics on which reviewer skill development could be supported as they seek to improve the research funding review process and evaluate outcomes. Coveney et al [12] and Turner, Bull, Chinnery, Hinks, Mcardle, Moran, Payne, Woodford Guegan, Worswick, and Wyatt [20], conducted qualitative studies capturing first person reports from peer review panelists concerning the peer review process, fairness, and the criteria used in decision-making. While skills can be extrapolated from Coveney et al [12] (group dynamics), Mow [13] (definitions of excellence, interaction), Turner et al [20] (time, good reviewer, value), and Bol [21] (writing, using tools); only Porter [22] (skimming, big picture, discernment), Member [23] (how to prepare, one's role, utilizing program guidelines), and Irwin, Gallo, and Glisson [24] (efficiency, writing, decision-making, evaluation) explicitly discuss reviewers' perspectives on panelist skill(s).…”

Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process

“…Although the new NSF process resulted in a reduction in the overall success rate it increased the success rate for the second stage full applications [6]. While there are generally few studies on the overall research application assessment processes a related area, looked at in some detail, is the value of external peer review [7][8][9][10] where previous research has examined the optimal numbers and expertise required. Guthrie et al point to the lack of evidence on the efficiency of peer review, where there may be biases in the systems used by funders and substantial burden to those involved, notably applicants [11].…”