Laughter and the Chair: Social Pressures Influencing Scoring During Grant Peer Review Meetings

Abstract

INTRODUCTIONDuring NIH peer review meetings (Bstudy sections^), scientists discuss and assign Bpriority scores^to grant applications that largely determine funding outcomes. Although the final priority score is an average of each panelist's score, their individual score is anchored to the scores declared publicly by those scientists (usually three) assigned to review and report on the grant application in detail. We have identified Bscore calibration talk^(SCT), a discourse practice where a study section member discusses and interprets the scoring rather than the content of a grant application. We found two forms: self-initiated SCT, when a panelist provides commentary about their own scoring (e.g., BSo I gave it a four, which was probably generous^); and other-initiated SCT, when a panelist challenges the scoring of an assigned reviewer (e.g., BYeah, that was generous.^). Only other-initiated SCT correlated with changes from the initial to the final score among the assigned reviewers of NIH R01 applications. 1 To gain insights into which interactional patterns accompanying SCT influence score change, we examined in detail the five cases of SCT followed by immediate declaration of a score change. METHODSMeticulously adhering to NIH practices, we constructed and video-recorded four study sections where groups of 8-12 oncology researchers evaluated the same 25 R01 grant applications previously submitted to NIH. Pier et al. provides a full description of our methodology. 1 Participant-reviewers evaluated R01 applications previously reviewed by study sections within NIH's National Cancer Institute. Applications were donated by Principal Investigators identified using NIH's public access database, then de-identified and re-identified by the research team. All applications had been funded either on the first submission or after revision between October 1, 2012 and September 31, 2015.We transcribed all discourse with attention to verbal and non-verbal actions, timing of pauses, marking of overlaps, and details of sound production, including laughter. We identified the speaker who initiated SCT, the assigned reviewer who was the target, and whether or not the target announced a score change. We coded and analyzed data as a case series, 2 treating score change as the Bconditionû nder study and calculating the odds ratio-given Bexposure^to identified patterns of discourse associated with SCT. 2 RESULTSWe identified 15 cases of other-directed SCT, five featuring immediate, public score change (Table 1). Each announced score change was preceded either by shared laughter among study section members (N = 4) or SCT by the chair of the study section (N = 1). There were no cases of immediate score change when a panelist initiated SCT but no shared laughter ensued (7/15). No instances of score change in conjunction with SCT occurred without either laughter or the chair's initiation of SCT.Odds ratios indicated that score change was significantly more likely when laughter was present during SCT than when it was absent (OR = 16; 95%...

A new approach to grant review assessments: score, then rank

“…Collecting the ranking data at the end of all proposal discussion has an added benefit, as it may help to mitigate any temporal effects, as sometimes the decision-making norms of a panel can evolve over the length of the review meeting [ 55 , 56 ]. For example, the panel may reach agreement on how to interpret the review criteria only after discussion at the panel meeting [ 57 ].…”

Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion

“…Further, the effectiveness of discussion from less persuasive reviewers may be hindered by a passive chair compared to a more engaged and assertive chair. Previous research has reported the importance of score-calibration comments and even laughter in the effectiveness of panel discussion, although it is unclear if these are affected by chair facilitation [8,20,21]. Future studies should include a focus on the social influences and group dynamics between panel reviewers, informed by the literature on small group decision making in other contexts.…”

Grant reviewer perceptions of the quality and effectiveness of panel discussion

“…It may also be that reviewers are overconfident in the effectiveness of panel discussion, potentially because they were directly involved in this discussion (Moore and Healey, 2007). Future studies should examine actual panel discussions and analyze for potential linguistic and stylistic differences in FTF and Vcon/Tcon panels that may help explain the previously reported scoring differences (Raclaw 2017, Pier 2019. It would also be interesting to gather perceptions from outside panel observers (like scientific review officers who manage panels for funding agencies) which may counter reviewer perceptions, as well as compare reviewer survey responses to actual scoring data, both of which may serve to assess for overconfidence.…”

“…Also, are more persuasive reviewers hindered more by review format than less proactive reviewers? Some have reported the importance of score-calibration comments and even laughter in the effectiveness of panel discussion, although it is unclear if these are affected in any way by review format (Pier et al, 2019). And as discussion has traditionally affected the funding status of only a small proportion of proposals (Martin 2010, Fogelholm 2012, Carpenter 2015, these types of studies should be examined in parallel with those examining the decision making processes that occur at the individual reviewer level.…”