The effect of publishing peer review reports on referee behavior in five scholarly journals

Abstract

To increase transparency in science, some scholarly journals are publishing peer review reports. But it is unclear how this practice affects the peer review process. Here, we examine the effect of publishing peer review reports on referee behavior in five scholarly journals involved in a pilot study at Elsevier. By considering 9,220 submissions and 18,525 reviews from 2010 to 2017, we measured changes both before and during the pilot and found that publishing reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around times. Younger and non-academic scholars were more willing to accept to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report. These findings suggest that open peer review does not compromise the process, at least when referees are able to protect their anonymity.

Author-Reviewer Homophily in Peer Review

“…Future work is necessary to confirm 675 and expand upon our findings, assess the extent to which they can be generalized, establish 676 causal relationships, and mitigate the effects of these methodological limitations. To aid in this 677 effort, we have made as much as possible of the data and analysis publicly available at: more open peer review did not compromise the integrity or logistics of the process, so long as 729 reviewers could maintain anonymity [124]. 730 Other alternatives to traditional peer review have also been proposed, including study 731 pre-registration, consultative peer review, and hybrid processes (eg: [58,[125][126][127][128][129]), as well as 732 alternative forms of dissemination, such as preprint servers (e.g., arXiv, bioRxiv) which have in 733 recent years grown increasingly popular [130].…”

A tale of two 'opens': intersections between Free and Open Source Software and Open Scholarship

“…This represents an unusual implication, as just because something has been 'peer reviewed' or not, does not mean that it should be treated any more or less critically as a responsible method of scholarship; especially when that review process remains concealed. Only recently has any progress been made in measuring the quality of peer review and reviewers themselves (Bianchi, Grimaldo, and Squazzoni 2019), aided by some publishers opening up their data on peer review (Bravo et al 2019). However, given that critical information about peer review is typically kept private by publishers, it is hard to gauge reviewer and review quality, and yet curiously its use remains almost ubiquitous and held in high-regard (Callaham and Tercier 2007).…”

Ten myths around open scholarly publishing

“…But these subtleties are lost on the general public, who often only hear the myth that published in a journal with peer review is the "gold standard" and can erroneously equate published research with the truth. Thus, more care must be taken over how peer review, and the results of peer reviewed research, are communicated to non-specialist audiences; particularly during a time in which a range of technical changes and a deeper appreciation of the complexities of peer review are emerging (Bravo et al 2019;Tennant 2018;Squazzoni, Grimaldo, and Marušić 2017;Allen et al 2018). This will be needed as the scholarly publishing system has to confront wider issues such as retractions (Budd, Sievert, and Schultz 1998b;Fang and Casadevall 2011;Moylan and Kowalczuk 2016) and replication or reproducibility 'crisis' (Collaboration 2015;Munafò et al 2017a;Fanelli 2018).…”