Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial

Abstract

Objectives To see whether telling peer reviewers that their signed reviews of original research papers might be posted on the BMJ's website would affect the quality of their reviews. Design Randomised controlled trial. Setting A large international general medical journal based in the United Kingdom. Participants 541 authors, 471 peer reviewers, 12 editors. Intervention Consecutive eligible papers were randomised either to have the reviewer's signed report made available on the BMJ's website alongside the published paper (intervention group) or to have the report made available only to the author-the BMJ's normal procedure (control group). The intervention was the act of revealing to reviewers-after they had agreed to review but before they undertook their review-that their signed report might appear on the website. Main outcome measures The main outcome measure was the quality of the reviews, as independently rated on a scale of 1 to 5 using a validated instrument by two editors and the corresponding author. Authors and editors were blind to the intervention group. Authors rated review quality before the fate of their paper had been decided. Additional outcomes were the time taken to complete the review and the reviewer's recommendation regarding publication. Results 558 manuscripts were randomised, and 471 manuscripts remained after exclusions. Of the 1039 reviewers approached to take part in the study, 568 (55%) declined. Two editors' evaluations of the quality of the peer review were obtained for all 471 manuscripts, with the corresponding author's evaluation obtained for 453. There was no significant difference in review quality between the intervention and control groups (mean difference for editors 0.04, 95% CI −0.09 to 0.17; for authors 0.06, 95% CI −0.09 to 0.20). Any possible difference in favour of the control group was well below the level regarded as editorially significant. Reviewers in the intervention group took significantly longer to review (mean difference 25 minutes, 95% CI 3.0 to 47.0 minutes). Conclusion Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ's website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.

What is open peer review? A systematic review

“…Opponents counter this by arguing that signing will lead to poorer reviews, as reviewers temper their true opinions to avoid causing offence. To date, studies have failed to show any great effect in either direction ( McNutt et al , 1990; van Rooyen et al , 1999; van Rooyen et al , 2010). However, since these studies derive from only one disciplinary area (medicine), the results cannot taken as representative and hence further research is undoubtedly required.…”

“…Van Rooyen and her colleagues found that open reports correlate with higher refusal rates amongst potential reviewers, as well as an increase in time taken to write review but no concomitant effect on review quality ( van Rooyen et al , 2010). Nicholson and Alperin’s small survey, however, found generally positive attitudes: “researchers … believe that open review would generally improve reviews, and that peer reviews should count for career advancement” ( Nicholson & Alperin, 2016).…”

Peer review in medical journals: Beyond quality of reports towards transparency and public scrutiny of the process

“…According to its detractors, open review may thus result in worse reports compared 137 to blind review, but this has not been observed in randomized, controlled trials [10,11,27]. 138…”

A Systematic Review of Peer Review for Scientific Manuscripts

“…The most prevalent theme was ethics [1,4,7,8,10,11,18,20,25,26,28,30,31,33,[40][41][42]46]. This theme occurred in 18 (49%) of the articles reviewed.…”

“…Fabricating results is ethical misconduct, but authors must also understand that altering data to misrepresent [11,20,31]. Another prevalent theme in the literature was the discussion of different versions of peer review, which was discussed in 12 (32%) of the articles reviewed [2,10,13,19,23,28,29,39,41,44,46,47]. These different methods of peer review include the single-blinded system, doubleblinded system, open system, peer agreement system, and author suggestion-based system ( Table 2).…”