A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels

Abstract

ObjectiveWith the use of teleconferencing for grant peer-review panels increasing, further studies are necessary to determine the efficacy of the teleconference setting compared to the traditional onsite/face-to-face setting. The objective of this analysis was to examine the effects of discussion, namely changes in application scoring premeeting and postdiscussion, in these settings. We also investigated other parameters, including the magnitude of score shifts and application discussion time in face-to-face and teleconference review settings.DesignThe investigation involved a retrospective, quantitative analysis of premeeting and postdiscussion scores and discussion times for teleconference and face-to-face review panels. The analysis included 260 and 212 application score data points and 212 and 171 discussion time data points for the face-to-face and teleconference settings, respectively.ResultsThe effect of discussion was found to be small, on average, in both settings. However, discussion was found to be important for at least 10% of applications, regardless of setting, with these applications moving over a potential funding line in either direction (fundable to unfundable or vice versa). Small differences were uncovered relating to the effect of discussion between settings, including a decrease in the magnitude of the effect in the teleconference panels as compared to face-to-face. Discussion time (despite teleconferences having shorter discussions) was observed to have little influence on the magnitude of the effect of discussion. Additionally, panel discussion was found to often result in a poorer score (as opposed to an improvement) when compared to reviewer premeeting scores. This was true regardless of setting or assigned reviewer type (primary or secondary reviewer).ConclusionsSubtle differences were observed between settings, potentially due to reduced engagement in teleconferences. Overall, further research is required on the psychology of decision-making, team performance and persuasion to better elucidate the group dynamics of telephonic and virtual ad-hoc peer-review panels.

Grant reviewer perceptions of the quality and effectiveness of panel discussion

“…Interestingly, we found no significant differences in reported discussion facilitation between FTF and Vcon/Tcon review formats ( Table 3). While we and others have previously found subtle differences in scoring and the length of discussion times between Tcon and FTF settings (Carpenter et al, 2015, Pier et al, 2015, this doesn't seem to affect reviewer perceptions of how the discussion was facilitated, although the differences found between FTF and Wb settings underscores the importance of at least audio-facilitated communication and discussion.…”

“…The literature surrounding grant peer review as a whole is very limited, and while some studies in the literature have examined review panel discussion and its effects on scoring (Obrecht 2007, Martin, 2010, Fogelham 2012, Fleurence 2014, Forsythe 2018), only four have contrasted traditional and alternate review formats (Gallo 2013, Carpenter 2015, Pier 2015, Vo and Trocki 2015. While Gallo et al (2013) has found no significant differences between face-to-face (FTF) or teleconference (Tcon) panels in terms of the average, breadth or levels of contentiousness in the final scores, both Pier et al (2015) and Carpenter et al (2015) noted that the effect of discussion on scoring (shifts in scoring by assigned reviewers after discussion) was slightly but significantly muted in Tcon panels as compared to FTF panels (Vcon panels in the case of Pier et al). Consistent with previous findings, these analyses found the magnitude of these scoring shifts after discussion were small and only affected the funding status of a small portion of grants.…”

External Tests of Peer Review Validity Via Impact Measures

“…Thus, there is a need to look at the internal validity of the review process with examinations of potential reviewer bias, review structures and baselines of decision making (Magua et al, 2017). These types of internal tests of review process validity are not included in this manuscript, but are crucial for assessing other expectations of peer review (Wood and Wessely, 2003), like fairness (Lee et al, 2013), efficiency (Carpenter et al, 2015) and rationality (Gallo et al, 2016).…”

Peer Review Practices for Evaluating Biomedical Research Grants: A Scientific Statement From the American Heart Association

“…Carpenter et al 16 compared the effect of discussion in teleconference and face-to-face peer review panels for an NIH R01-like program offered by a US federal agency from 2009 to 2012. They found that the effect of discussion was small, on average, for both types of panels.…”

“…Carpenter et al 16 also noted that the major differences between teleconference and face-to-face panels are the virtual versus in-person communication and the level of trust among reviewers. Trust forms through visual cues and the ability to socialize during breaks and meals.…”