Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion

Abstract

Background: Funding agencies have long used panel discussion in the peer review of research grant proposals as a way to utilize a set of expertise and perspectives in making funding decisions. Little research has examined the quality of panel discussions and how effectively they are facilitated. Methods: Here, we present a mixed-method analysis of data from a survey of reviewers focused on their perceptions of the quality, effectiveness, and influence of panel discussion from their last peer review experience. Results: Reviewers indicated that panel discussions were viewed favorably in terms of participation, clarifying differing opinions, informing unassigned reviewers, and chair facilitation. However, some reviewers mentioned issues with panel discussions, including an uneven focus, limited participation from unassigned reviewers, and short discussion times. Most reviewers felt the discussions affected the review outcome, helped in choosing the best science, and were generally fair and balanced. However, those who felt the discussion did not affect the outcome were also more likely to evaluate panel communication negatively, and several reviewers mentioned potential sources of bias related to the discussion. While respondents strongly acknowledged the importance of the chair in ensuring appropriate facilitation of the discussion to influence scoring and to limit the influence of potential sources of bias from the discussion on scoring, nearly a third of respondents did not find the chair of their most recent panel to have performed these roles effectively. Conclusions: It is likely that improving chair training in the management of discussion as well as creating review procedures that are informed by the science of leadership and team communication would improve review processes and proposal review reliability.

Grant Review Feedback: Appropriateness and Usefulness

“…Our results are limited by a relatively low response rate, although this response rate approximates the rate of similar surveys on peer review (Gallo et al 2020a;Ware 2008). The majority of funding agency comments in our survey mentioned the NIH as a recent source of review feedback; the gender, race, and degrees of our sample are similar to those reported from surveys and analyses of NIH applicants (NIH 2012a;Ginther et al 2011Ginther et al , 2016.…”

“…Given the paucity of data surrounding grant review feedback, particularly from an applicant’s perspective, we created a survey for research scientists that asked applicant respondents to rate and comment on the review feedback they received on their most recent submission. Three publications have resulted from this survey (Gallo et al 2018 ; Gallo et al 2020a , b ), but none of them focused on the questions related to the usefulness and appropriateness of peer review feedback; these questions are now addressed in the proceeding analysis.…”

The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF2021

“…Studies show no significant difference in the quality of the decision making in teams using written (text-only) or audio-only communication (Gallo et al 2013), but do show a benefit of adding video formats (including videoconferencing facilities) resulting in a significant improvement in the quality of team's deliberations and resultant strategic decision making (Baker 2002). Comparative research on the scoring patterns of peer reviews of traditional academic criteria has found only subtle differences in those between F2F and virtual panels; however, there was a decrease in discussion quality when deliberations were conducted virtually (Carpenter et al 2015;Gallo, Carpenter, and Glisson 2013;Gallo et al 2020). This is the case regardless of whether virtual deliberation is conducted using video conferencing, instant messages or other supported web technologies (such as parallel chat functions) (Carpenter et al 2015;Gallo, Carpenter, and Glisson 2013;Gallo et al 2020;Pier et al 2017).…”

“…Comparative research on the scoring patterns of peer reviews of traditional academic criteria has found only subtle differences in those between F2F and virtual panels; however, there was a decrease in discussion quality when deliberations were conducted virtually (Carpenter et al 2015;Gallo, Carpenter, and Glisson 2013;Gallo et al 2020). This is the case regardless of whether virtual deliberation is conducted using video conferencing, instant messages or other supported web technologies (such as parallel chat functions) (Carpenter et al 2015;Gallo, Carpenter, and Glisson 2013;Gallo et al 2020;Pier et al 2017). Thus, regardless of the sophistication of web-technologies to attempt to recreate the benefits associated with F2F, F2F remains the preferred, and more efficient mechanism to support complex decision-making, with this effect amplified for new panel groups (O'Neill et al 2016).…”

Grant Review Feedback: Appropriateness and Usefulness

“…Given the paucity of data surrounding grant review feedback, particularly from an applicant's perspective, we created a survey for research scientists that asked applicant respondents to rate and comment on the review feedback they received on their most recent submission. Three publications have resulted from this survey (Gallo et al 2018, Gallo et al 2020a, Gallo et al 2020b), but none of them focused on the questions related to the usefulness and appropriateness of peer review feedback; these questions are now addressed in the proceeding analysis.…”