‘Your comments are meaner than your score’: score calibration talk influences intra- and inter-panel variability during scientific grant peer review

Abstract

In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process. HHS Public AccessAuthor manuscript Res Eval. Author manuscript; available in PMC 2018 January 01.Published in final edited form as:Res Eval. 2017 January ; 26(1): 1-14. doi:10.1093/reseval/rvw025. Author Manuscript Author ManuscriptAuthor Manuscript Author ManuscriptKeywords peer review; discourse analysis; decision making; collaboration As the primary means by which scientists secure funding for their research programs, grant peer review is a keystone of scientific research. The largest funding agency for biomedical, behavioral, and clinical research in the USA, the National Institutes of Health (NIH), spends more than 80% of its $30.3 billion annual budget on funding research grants evaluated via peer review (NIH 2016). As part of the mechanism by which this money is allocated to scientists, collaborative peer review panels of expert scientists (referred to as 'study sections' at NIH) convene to evaluate grant applications and assign scores that inform later funding decisions by NIH governance. Thus, deepening our understanding of how peer review ostensibly identifies the most promising, innovative research is crucial for the scientific community writ large. The present study builds upon existing work evaluating the reliability of peer review by examining how the discourse practices of reviewers during study section meetings may contribute to low reliability in peer review outcomes.The NIH peer review process is structured around study sections that engender distributed expertise (Brown et al. 1993), as reviewers evaluate applications based on their particular domain(s) of expertise but then share their specialized knowledge with others who have related but distinct expertise. The very structure of study sections thus facilitates what Brown and colleagues (1993) identify as mutual appropriation among grou...

Laughter and the management of divergent positions in peer review interactions

“…Indeed we find that monitoring of the scoring patterns can motivate explicit “policing,” whereby individual meeting members call into question the legitimacy of a score, and by extension, the legitimacy of an assigned reviewer’s own expertise (Pier et al, 2017). The scoring process is thus of immediate consequence to the social and professional standings of the meeting participants as they construct their professional expertise in situ during the meeting.…”

“…Research on grant peer review panel interactions has almost exclusively relied on ethnographic interviews and participant observation (Ahlqvist et al, 2013; Lamont, 2009) rather than close analysis of unfolding interaction. Indeed, research directly examining discourse and interaction in review panel meetings has only recently emerged (Gallo, Carpenter, & Glisson, 2013; Pier, Raclaw, Kaatz, Carnes, Nathan, & Ford, 2017; Authors, 2015). As part of a larger project aimed at documenting interactional practices inside the “black box” of peer review meetings, 1 in this paper we focus on how participants in these settings use laughter as a resource to manage the divergence of evaluative positions that characterizes the give and take of joint grant evaluation.…”

“… 2 See Pier et al (2015) for a full description of the design and implementation of the peer review panels examined in this study. …”

“… 3 Reviewers are provided with these details of the scoring rubric prior to the grant review period, though there is evidence that reviewers’ individual understanding of these scores are fluid and subjective, and during meetings reviewers actively work toward negotiating shared understandings of the “meaning” of a score (Pier et al, 2015). …”

Laughter and the Chair: Social Pressures Influencing Scoring During Grant Peer Review Meetings

“…Pier et al provides a full description of our methodology. 1 Participant-reviewers evaluated R01 applications previously reviewed by study sections within NIH's National Cancer Institute. Applications were donated by Principal Investigators identified using NIH's public access database, then de-identified and re-identified by the research team.…”