Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature

Abstract

Preprint usage is growing rapidly in the life sciences; however, questions remain on the relative quality of preprints when compared to published articles. An objective dimension of quality that is readily measurable is completeness of reporting, as transparency can improve the reader's ability to independently interpret data and reproduce findings. In this observational study, we initially compared independent samples of articles published in bioRxiv and in PubMed-indexed journals in 2016 using a quality of reporting questionnaire. After that, preprints were also compared to their own journalpublished versions. Peer-reviewed articles had, on average, higher quality of reporting than preprints, although the difference was small, with absolute differences of 5.0 % [95% CI 1.4, 8.6] and 4.7 % [95% CI 2.4, 7.0] of reported items in the independent samples and paired sample comparison, respectively. There were larger differences favoring peerreviewed articles in subjective ratings of how clearly titles and abstracts presented the main findings and how easy it was to locate relevant reporting information. Changes in reporting from preprints to peer-reviewed versions did not correlate with the impact factor of the publication venue or with the time lag from bioRxiv to journal publication. Overall, our results suggest that editorial peer review has a statistically significant but small impact on improving quality of reporting.They also show that quality of reporting in preprints in the life sciences is within a similar range as that of peer -reviewed articles, albeit slightly lower on average, supporting the idea that preprints should be considered valid scientific contributions.

Preprinting the COVID-19 pandemic

“…2) suggesting that, generally, preprints were relatively of good quality. Furthermore, recent studies have suggested that the quality of reporting in preprints differs little from their later peer-reviewed articles [44] and we ourselves are currently undertaking a more detailed analysis (see version 1 of our preprint for an initial analysis of published COVID preprints [45]). However, the problem of poor-quality science is not unique to preprints and ultimately, a multi-pronged approach is required to solve some of these issues.…”

The limitations to our understanding of peer review

“…However, this does not mean that preprints are the solution to all issues around peer review and scholarly publishing, especially as they are still regarded in different ways by different communities; something that undoubtedly requires further study [99]. With the recent explosion of preprints in the Life Sciences [103], a number of different services have emerged that 'overlay' peer review in one form or another on top of the developing preprint infrastructure [104], for example, biOverlay in the Life Sciences. However, the general uptake of such services appears to be fairly low [105]; most recently, this led to Academic Karma, a leading platform in this area, to shut-down (April 2019).…”

A systematic examination of preprint platforms for use in the medical and biomedical sciences setting

“…Empirical evidence to support the use of editors and peer reviewers as a mechanism to ensure the quality of biomedical research is relatively weak [21, 22] although other studies have rendered peer review as being potentially useful [23, 24]. This review provides some justification that preprint platforms might be a reasonable option for researchers, especially given the time spent and associated cost of peer review [25].…”