Mapping the content of comments on bioRxiv and medRxiv preprints

Abstract

Introduction: Preprints have been increasingly used in biomedical sciences, providing the opportunity for research to be publicly assessed before journal publication. With the increase in attention over preprints during the COVID-19 pandemic, we decided to assess the content of comments left on preprint platforms. Methods: Preprints posted on bioRxiv and medRxiv in 2020 were accessed through each platform's API, and a random sample of preprints that had received between 1 and 20 comments was analyzed. Comments were evaluated in triplicate by independent evaluators using an instrument that assessed their features and general content. Results: 7.3% of preprints received at least 1 comment during a mean follow-up of 7.5 months. Analyzed comments had a median size of 43 words. Criticisms, corrections or suggestions were the most prevalent type of content, followed by compliments or positive appraisals and questions. Most critical comments regarded interpretation, data collection and methodological design, while compliments were usually about relevance and implications. Conclusions: Only a small percentage of preprints posted in 2020 in bioRxiv and medRxiv received comments in these platforms. When present, however, these comments address content that is similar to that analyzed by traditional peer review. A more precise taxonomy of peer review functions would be desirable to describe whether post-publication peer review fulfills these roles.

Characterization and Reach of Orthopaedic Research Posted to Preprint Servers: Are We “Undercooking” Our Science?

“…Furthermore, we used three peer-reviewed publication databases to verify that a publication corresponds to a specific preprint posting, but it can still be difficult to ascertain that two articles are describing the exact same study in a small number of cases. Although medRxiv and bioRxiv state that the process of updating preprinted articles with publication information is automatic, authors most often used comments on bioRxiv and medRxiv preprints to provide updates on the preprint’s publication status [12], suggesting this type of update relies on preprint authors. On Research Square, the journal-integrated service relies on preprint authors to opt in, and the peer review timeline depends on what the participating journals allow to be shown.…”

“…Current screening measures in preprint media may be insufficient, because a cross-sectional analysis of author instructions on 57 preprint servers demonstrated inadequate screening and recommendations for risk-of-bias reporting, data transparency, or research integrity guidelines [31]. Given that fewer than 10% of preprint articles receive comments, most of this group only had a single comment, and approximately 11% of the comments originated from an author of the preprint article itself [12], it is evident that preprints receive far less scrutiny than articles under peer review. Although proponents of preprints in clinical sciences emphasize the importance of rapid dissemination for scientific exchange [25], pathways through the peer-review process are available through quality journals for emergency circumstances, as demonstrated by a timely Coronavirus-19 publication in the New England Journal of Medicine in January 2020 [46].…”

“…The impact of preprints related to Coronavirus-19 may not be generalizable to orthopaedic preprinted articles, because this study did not evaluate the quality of the research presented in preprinted articles. Nonetheless, preprinting has caused serious harms [16,40], and most preprints lack scrutiny [12]. It remains unclear whether growing metrics of preprinted articles correlate with changes in the quality of care of orthopaedic patients.…”

Promoting diversity and overcoming publication barriers in Latin American neuroscience and Alzheimer's disease research: A call to action

“…The COVID‐19 pandemic further revealed the pressing need for open and fast science progress. Preprint platforms comprise an alternative for open access to scientific knowledge by LMICs, though reflections on the absence of peer review and the quality of reporting and content of the published material have been discussed in the literature 57–59 …”

“…Preprint platforms comprise an alternative for open access to scientific knowledge by LMICs, though reflections on the absence of peer review and the quality of reporting and content of the published material have been discussed in the literature. [57][58][59] Therefore, pay-walling practices represent a bottleneck for publication and dissemination of findings that could stimulate further research, collaborative work, and science advancement in LMICs. For instance, these publishing drawbacks reduce the traction and hinder competitiveness of LMIC research groups for international grant schemes that, in turn, discourages further research.…”