The collective wisdom in the COVID-19 research: comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles

Wang, Yuejiao; Cao, Zhidong; Zeng, Daniel; Zhang, Qingpeng; Luo, T. David

doi:10.1101/2020.07.22.20160291

Cited by 3 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The semantic analysis of the title, abstract, and body of the preprints in arXiv and bioRxiv found little notable difference with their peer-reviewed versions (44). Comparison of preprints and their peer-reviewed versions on COVID-related research also found coherence in the reported results (41, 44, 46), extending the findings of these studies. In addition, analysis of the primary data constituting the evidence base suggests that estimate values in published papers are nearly identical to those in preprint versions (24, 25).…”

Section: Discussionsupporting

confidence: 76%

Predicting causal citations without full text

Hoppe

Arabi

Hutchins

2022

Preprint

View full text Add to dashboard Cite

Citation analysis generally assumes that each citation documents causal knowledge transfer that informed the conception, design, or execution of the main experiments. Citations may exist for other reasons. In this paper we identify a subset of citations that are unlikely to represent causal knowledge flow. Using a large, comprehensive feature set of open access data, we train a predictive model to identify such citations. The model relies only on the title, abstract, and reference set and not the full-text or future citations patterns, making it suitable for publications as soon as they are released, or those behind a paywall. We find that the model identifies, with high prediction scores, citations that were likely added during the peer review process, and conversely identifies with low prediction scores citations that are known to represent causal knowledge transfer. Using the model, we find that federally funded biomedical research publications represent 30% of the estimated causal knowledge transfer from basic studies to clinical research, even though these comprise only 10% of the literature, a three-fold overrepresentation in this important type of knowledge transfer. This finding underscores the importance of federal funding as a policy lever to improve human health.

show abstract

Section: Discussionsupporting

confidence: 76%

Predicting causal citations without full text

Hoppe

Arabi

Hutchins

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Third, for both high-and low-quality preprints, the checklist highlighted issues with transparency and research integrity. This mirrors the state of the COVID-19 literature, as we have seen that though the credibility of the research reported in preprints can be sound (32)(33)(34), data and materials sharing may nonetheless often be lacking (18,19). This could be taken as a weakness of the checklist, as this item is not particularly discriminative of preprint quality.…”

Section: Discussionsupporting

confidence: 57%

“…Preprints on clinical trials for example were shown to report results that may be just as credible as those of publications (32). This finding is reinforced by comparisons between preprints and publications on metrics like the basic reproduction number, transmissibility, incubation period, and infectious period, showing no evidence for a difference in the results presented in peer-reviewed vs. non-peer-reviewed literature (33,34). Nonetheless, a large portion of COVID-19 preprints show substantial changes in methods and results after peer-review (nearly half of the preprints analysed by Oikonomidi (2020) and Nicolalde et al (2020) (25,27)), suggesting flaws in the most essential elements of many COVID-19 preprints.…”

Section: Constructing and Validating Precheck: A Checklist To Evaluat...mentioning

confidence: 99%

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Turoman¹,

Heyard²,

Schwab³

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: The quality of COVID-19 preprints should be considered with great care, as we have seen that their contents can influence public policy. Efforts to improve preprint quality have mostly focused on introducing quick peer review, but surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The aim of the PRECHECK project was to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and in general.Methods: To create a checklist, we applied a 4-step procedure consisting of an initial internal review, followed by an external review by a pool of experts methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and an implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. Additionally, after each internal review round, we applied the checklist on a set of high-quality preprints (i.e., from an online list of milestone research works in the COVID-19 pandemic) and low-quality preprints (i.e., that were eventually retracted) to verify whether the checklist can discriminate between the two categories.Results: At the external review step, of 54 experts contacted, 26 responded. The final checklist contained 4 elements (Research question, Study type, Transparency and integrity, and Limitations), with ‘superficial’ and ‘deep’ levels at which preprints can be evaluated. When using both levels of evaluation, the checklist was effective at discriminating high- from low-quality preprints. Its usability was confirmed in workshops with members of our target audience: Bachelors students in Psychology and Medicine, as well as with science journalists.Conclusions: We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind. We thus believe that our checklist has great potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this may very well extend beyond COVID-19.

show abstract

“…Studies suggest that the discrepancy between the preprints and peer-reviewed articles is minor and the quality of reporting is within comparable range, advocating in favor of sharing research findings in preprints (19, 21). Wang et al also compared epidemiological estimates for preprints and published articles (separate articles, rather than different versions of the same papers as studied here), and found that basic reproduction number, incubation period, and infectious period were similar in preprints and peer-reviewed articles (24).…”

Section: Discussionmentioning

confidence: 89%

“…As discoveries are posted to preprint servers, disclaimers have been added to emphasize that the paper has not been peer reviewed (16). However, as noted above, little is known about how the extent to which data in biomedical preprints systematically differ from their peer-reviewed counterfactuals, despite the importance of this emerging question (17)(18)(19)(20)(21)(22)(23)(24). This knowledge gap is concerning not only for navigating preprints in this infectious disease outbreak, but also for navigating future emergencies.…”

Section: Introductionmentioning

confidence: 99%

Robustness of evidence reported in preprints during peer review

Nelson

Schwenn

et al. 2022

Preprint

View full text Add to dashboard Cite

Adoption of preprints dramatically expanded during the COVID-19 pandemic. Many have expressed concern that the risk of flawed decision-making is increased by relying on preprint data that would not survive peer review. We therefore asked how much the information presented in preprints is expected to change after review. We quantify attrition dynamics of over 1000 epidemiological estimates first reported in 100 matched preprints studying COVID-19. We find that 89% of point estimates persist through peer review. Of these, the correlation between preprint and published estimate values is extremely high at 0.99, and there is no systematic trend toward estimate inflation or deflation during review. A higher degree of data alteration during peer review, either in terms of magnitude or deletion, might be expected in papers never published because of their lower quality, which could limit the generalizability of our results. Importantly, we find that expert peer review scores of preprint quality are not related to eventual publication in a peer reviewed journal, mitigating this concern. Uncertainty is reduced somewhat, however, as authors add another 18% of data points compared to the preprint version. Confidence interval ranges also decrease by a small but statistically significant 7%. Therefore, the evidence base presented in preprints is highly stable, and where data change during review, uncertainty is expected to decrease by a small amount on average. These results lend credence to the use of preprints, as one component of the biomedical research literature, in decision-making. These results can help inform the use of preprints during the ongoing pandemic as well as future disease outbreaks.

show abstract

The collective wisdom in the COVID-19 research: comparison and synthesis of epidemiological parameter estimates in preprints and peer-reviewed articles

Cited by 3 publications

References 21 publications

Predicting causal citations without full text

Predicting causal citations without full text

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Robustness of evidence reported in preprints during peer review

Contact Info

Product

Resources

About