Prevalence of short peer reviews in leading general medical journals: a study of peer-review length at The BMJ, PLOS Medicine, and BMC Medicine

Abstract

Background: High-quality peer reviews are often thought to be essential to ensuring the integrity of the scientific publication process but measuring peer review quality is challenging. Although imperfect, review word count could serve as a simple, objective metric of review quality. We aimed to examine the prevalence of very short reviews and how often they inform editorial decisions on research articles in leading general medical journals. Methods: We compiled a data set of peer reviews from published full-length original research articles in The BMJ, BMC Medicine, and PLOS Medicine for the years 2003 to 2022. In our primary analyses, we used a threshold of <200 words to calculate the prevalence of very short reviews. In secondary analyses, we also used thresholds of <100 and <300 words. In addition to disaggregating results by journal and year, we plotted the proportion of articles for which the first editorial decision was made based on a set of peer reviews in which very short reviews constituted 100%, >=50%, >=33%, or >=20% of the reviews. Results: In this sample of 11,466 reviews corresponding to 4,038 published articles, the median review word count was 425 (Interquartile Range=253-675), and the mean was 520 (Standard Deviation=401). The overall prevalence of very short (<200 words) reviews was 17.1% [95% CI: 16.4%-17.8%]. Across the three journals, 20.9% [95% CI: 19.6%-22.2%] of initial editorial decisions were based on review sets containing ≥50% very short reviews. The prevalence of very short reviews and share of editorial decisions based on review sets containing >=50% very short reviews was highest for BMC Medicine at 26.8% [95% CI: 25.1%-28.5%] and lowest for The BMJ at 7.3% [95% CI: 5.7%-8.9%]. Conclusions: A substantial proportion of initial editorial decisions for published articles in these three leading general medical journals was based on peer reviews of such short length that they were unlikely to be of high quality. Future research should determine whether monitoring peer review length is a useful approach to improving the quality of the peer review process and which interventions, such as incentives and norm-based interventions, are most effective in soliciting more detailed reviews.