Transparency in peer review: Exploring the content and tone of reviewers’ confidential comments to editors

Abstract

Purpose Recent calls to improve transparency in peer review have prompted examination of many aspects of the peer-review process. Peer-review systems often allow confidential comments to editors that could reduce transparency to authors, yet this option has escaped scrutiny. Our study explores 1) how reviewers use the confidential comments section and 2) alignment between comments to the editor and comments to authors with respect to content and tone. Methods Our dataset included 358 reviews of 168 manuscripts submitted between January 1, 2019 and August 24, 2020 to a health professions education journal with a single blind review process. We first identified reviews containing comments to the editor. Then, for the reviews with comments, we used procedures consistent with conventional and directed qualitative content analysis to develop a coding scheme and code comments for content, tone, and section of the manuscript. For reviews in which the reviewer recommended “reject,” we coded for alignment between reviewers’ comments to the editor and to authors. We report descriptive statistics. Results 49% of reviews contained comments to the editor (n = 176). Most of these comments summarized the reviewers’ impression of the article (85%), which included explicit reference to their recommended decision (44%) and suitability for the journal (10%). The majority of comments addressed argument quality (56%) or research design/methods/data (51%). The tone of comments tended to be critical (40%) or constructive (34%). For the 86 reviews recommending “reject,” the majority of comments to the editor contained content that also appeared in comments to the authors (80%); additional content tended to be irrelevant to the manuscript. Tone frequently aligned (91%). Conclusion Findings indicate variability in how reviewers use the confidential comments to editor section in online peer-review systems, though generally the way they use them suggests integrity and transparency to authors.

Peer Review, Confidential Comments to the Editor, and the Golden Rule

“…Compliments in one section followed by hidden criticism in the other are hard to reconcile with any degree of sincerity or good faith, let alone the golden rule. Nonalignment of the tone of the comments (9% of reviews) 9 does not conform to ethical guidelines, which call for nonconflicting comments 17,19 . Confidential comments are always more critical 9 .…”

“…Whether reviewers' identities should be confidential in the first place is at least open to question. The concern is that reviewers may not be forthcoming with honest reviews if the author is a person of influence in a specialty and can impact the reviewer's career 9 . However, the downside is that anonymous reviewers may make loose or derogatory comments that they would not make if their identity were known.…”

“…A goal of open peer review is to improve the quality of reviews and eliminate problems caused by nontransparency 25 . Some journals either require reviewers to share their identity or give them that option 9 . A recent study found that reviews for articles in journals with an open peer review policy were significantly less harsh than those with an anonymous review process 26 .…”

“…To better assess the value of confidential comments, it is useful to have data. Recently, O'Brien et al 9 published their study of 358 reviews of 168 manuscripts submitted to a health professions education journal with a single blind review process. Approximately half of the reviews (49%) included comments to the editor.…”

“…Although the Confidential Comments space is recommended for ethical breeches or conflicts of interest, these subjects are rarely raised (3%). 9 To withhold such important issues from the author, and let him or her suffer the consequences, would seem to violate the golden rule.…”