A New Approach for Timely Systematic Reviews of Rapidly Evolving Research: Identifying Evidence in Which We Can Place Our Confidence

Abstract

Coronavirus disease of 2019 (COVID-19) has impacted the world in unprecedented ways since first emerging in December 2019. In the last two years, the scientific community has made an enormous effort to understand COVID-19 and potential interventions. As of June 15, 2021, there were more than 140,000 COVID-19 focused manuscripts on PubMed and preprint servers, such as medRxiv and BioRxiv. Preprints, which constitute more than 15% of all manuscripts, may contain more up-to-date research findings compared to published papers, due to the sometimes lengthy timeline between manuscript submission and publication. Including preprints in systematic reviews and meta-analyses thus has the potential to improve the timeliness of reviews. However, there is no clear guideline on whether preprints should be included in systematic reviews and meta-analyses. Using a prototypical example of a rapid systematic review examining the comparative effectiveness of COVID-19 therapeutics, we propose including all preprints in the systematic review by assigning them a weight we term the "confidence score". Motivated by our observation that, unlike the traditional journal submission process which is unobserved, the timeline from submission to publication for a preprint can be observed and can be modeled as a time-to-event outcome. This observation provides a unique opportunity to model and quantify the probability that a preprint will be published, which can be used as a confidence score to weight preprints in systematic reviews and meta-analyses. To obtain the confidence score, we propose a novel survival cure model, which incorporates both the time from posting to publication for a preprint, and key characteristics of the study described in the content of the preprint. Using meta data from 158 preprints on evaluating therapeutic options for COVID-19 posted through 09/03/2020, we demonstrate the utility of the confidence score in weighting of preprints in a systematic review. Our proposed method has the potential to advance timely systematic reviews of the evidence examining COVID-19 and other clinical conditions with rapidly evolving evidence bases by providing an approach for inclusion of unpublished manuscripts.

Indian COVID-19 Preprints Submissions in bioRxiv and medRxiv Preprint Servers

“…One of the waysof creating new ideas isthe rapid production and dissemination of findings usually communicated through preprint servers or journals (Celi et al, 2021). As of June 25 2021, there were about 140,000 manuscripts on COVID-19 published or posted at PubMed, bioRxiv, and medRxiv (Tong et al, 2021), which indicates a new dawn of rapid evolution of research. Preprint server arXiv was the first server populated and popularized by physics, mathematics, and computer science community in 1991 (Vlasschaert et al, 2020).…”