Reliability of reviewers' ratings when using public peer review: a case study

Abstract

If a manuscript meets scientific standards and contributes to the advancement of science, it can be expected that two or more reviewers will agree on its value. Manuscripts are rated reliably when there is a high level of agreement between independent reviewers. This study investigates for the first time whether inter‐rater reliability, which is low with the traditional model of closed peer review, is also low with the new system of public peer review or whether higher coefficients can be found for public peer review. To investigate this question we examined the peer‐review process practiced by the interactive open access journal Atmospheric Chemistry and Physics (based on 465 manuscripts submitted between 2004 and 2006 receiving 1,058 reviews in total). The results of the study show that inter‐rater reliability is low (kappa coefficient) or reasonable (Intraclass Correlation Coefficient) in public peer review.

Understanding and supporting anonymity policies in peer review

“…Van Rooyen, Delamothe, and Evans (2010) found that telling reviewers that their signed reviews might be published (on the web) did not affect the quality of reviews, but it increased the time spent on composing reviews. Bornmann and Daniel (2010) analyzed public reviews submitted to the journal Atmospheric Physics and Chemistry and found that the level of inter-reviewer reliability was low, comparable to that of traditional peer-review processes. Bingham, Higgins, Coleman, and Van Der Weyden (1998) found that postpublication reviews by online readers can provide valuable feedback, but those reviews are often short and specific, and thus do not adequately replace full editorial peer review.…”

A multi-disciplinary perspective on emergent and future innovations in peer review

“…These studies typically use several different approaches to gather evidence on the functionality of peer review. Some, such as Bornmann & Daniel (2010b); Daniel (1993); Zuckerman & Merton (1971), used access to journal editorial archives to calculate acceptances, assess inter-reviewer agreement, and compare acceptance rates to various article, topic, and author features. Others interviewed or surveyed authors, reviewers, and editors to assess attitudes and behaviours, while others conducted randomized controlled trials to assess aspects of peer review bias ( Justice et al , 1998; Overbeke, 1999).…”

“…Rigorous, evidence-based research on peer review itself is surprisingly lacking across many research domains, and would help to build our collective understanding of the process and guide the design of ad-hoc solutions ( Bornmann & Daniel, 2010b; Bruce et al , 2016; Rennie, 2016; Jefferson et al , 2007). Such evidence is needed to form the basis for implementing guidelines and standards at different journals and research communities, and making sure that editors, authors, and reviewers hold each other reciprocally accountable to them.…”

Open peer review at four STEM journals: an observational overview

“…It has also been argued that OPR allows for easier identification of scientific misconduct ( Boldt, 2011), and that over time the quality of submitted articles will improve ( Hu et al , 2010; Prug, 2010). OPR affords referees the ability to gain credit for and cite their contributions to science communication ( Boldt, 2011; Bornmann & Daniel, 2010; Fitzpatrick, 2010; Prug, 2010; Pöschl, 2009). More broadly speaking, OPR provides the scholarly community an insight into author/referee conversations during the review process.…”