Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Gehanno, Jean-François; Grosjean, Julien; Darmoni, Stéfan Jacques; Rollin, Laétitia

doi:10.1101/2022.02.16.22271068

Cited by 4 publications

(3 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While it is true that many preprints never convert into publications (70-80% as of April 2021, see e.g., 25,26) and that unpublished preprints tend to be less cited than published ones (27,28), this is not necessarily due to poor quality. For one, the link between a preprint and its publication may be lost when it is the preprint that gets cited (29). Alternatively, the authors' decisions could be the cause, as some may avoid the publication process altogether (30).…”

Section: Constructing and Validating Precheck: A Checklist To Evaluat...mentioning

confidence: 99%

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Turoman¹,

Heyard²,

Schwab³

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: The quality of COVID-19 preprints should be considered with great care, as we have seen that their contents can influence public policy. Efforts to improve preprint quality have mostly focused on introducing quick peer review, but surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The aim of the PRECHECK project was to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and in general.Methods: To create a checklist, we applied a 4-step procedure consisting of an initial internal review, followed by an external review by a pool of experts methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and an implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. Additionally, after each internal review round, we applied the checklist on a set of high-quality preprints (i.e., from an online list of milestone research works in the COVID-19 pandemic) and low-quality preprints (i.e., that were eventually retracted) to verify whether the checklist can discriminate between the two categories.Results: At the external review step, of 54 experts contacted, 26 responded. The final checklist contained 4 elements (Research question, Study type, Transparency and integrity, and Limitations), with ‘superficial’ and ‘deep’ levels at which preprints can be evaluated. When using both levels of evaluation, the checklist was effective at discriminating high- from low-quality preprints. Its usability was confirmed in workshops with members of our target audience: Bachelors students in Psychology and Medicine, as well as with science journalists.Conclusions: We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind. We thus believe that our checklist has great potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this may very well extend beyond COVID-19.

show abstract

Section: Constructing and Validating Precheck: A Checklist To Evaluat...mentioning

confidence: 99%

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Turoman¹,

Heyard²,

Schwab³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Any further responses from the reviewers can be found at the end of the article Such initial findings are countered by a much more nuanced story of COVID-19 preprint quality. While it is true that many preprints never convert into publications 25,26 and that preprints tend to be less cited than the resulting peer reviewed publication, 27,28 this is not necessarily due to poor quality. The link between a preprint and its publication may be lost when it is the preprint that gets cited.…”

Section: Introductionmentioning

confidence: 99%

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Turoman,

Heyard,

Schwab

et al. 2024

F1000Res

View full text Add to dashboard Cite

Background The quality of COVID-19 preprints should be considered with great care, as their contents can influence public policy. Surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The PRECHECK project aimed to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and beyond. Methods To create a checklist, we applied a four-step procedure consisting of an initial internal review, an external review by a pool of experts (methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and a Preliminary implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. After each internal review round, we applied the checklist on a small set of high-quality preprints from an online list of milestone research works on COVID-19 and low-quality preprints, which were eventually retracted, to verify whether the checklist can discriminate between the two categories. Results At the external review step, 26 of the 54 contacted experts responded. The final checklist contained four elements (Research question, study type, transparency and integrity, and limitations), with ‘superficial’ and ‘deep’ evaluation levels. When using both levels, the checklist was effective at discriminating a small set of high- and low-quality preprints. Its usability for assessment and discussion of preprints was confirmed in workshops with Bachelors students in Psychology and Medicine, and science journalists. Conclusions We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind and facilitate discussions within, for example, a beginner-level lecture on research methods. We believe that our checklist has potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this extends beyond COVID-19.

show abstract

“…25 Such initial findings are countered by a much more nuanced story of COVID-19 preprint quality. While it is true that many preprints never convert into publications 25,26 and that preprints tend to be less cited than the resulting peer reviewed publication, 27,28 this is not necessarily due to poor quality. The link between a preprint and its publication may be lost REVISED Amendments from Version 1 In this updated version of the manuscript we have made the following changes: shortening the introduction while adding additional literature, updating the references, extending the Ethical considerations and Discussion sections to address specific reviewer comments regarding our workshops, creating a Limitations section, adding the introduction section that is in the checklist document into the manuscript itself (above the checklist in the manuscript in Table 2), renaming "sensitivity test" to "pilot testing" and "implementation phase" to "preliminary implementation phase" throughout, softening conclusions about the efficacy of the checklist to discern between high-and low-quality preprints, moving the section describing the workshop into the methods section and adding further detail on no data being collected for statistical analyses, adding to the Study design section that the preliminary implementation phase does not allow a systematic validation of the checklist, explaining the sampling rationale for workshop participants, and clarifying the expert group taxonomy.…”

Section: Introductionmentioning

confidence: 99%

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

et al. 2023

View full text Add to dashboard Cite

Background: The quality of COVID-19 preprints should be considered with great care, as their contents can influence public policy. Efforts to improve preprint quality have mostly focused on introducing quick peer review, but surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The PRECHECK project aimed to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and beyond. Methods: To create a checklist, we applied a four-step procedure consisting of an initial internal review, an external review by a pool of experts (methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and an implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. After each internal review round, we applied the checklist on a set of high-quality preprints from an online list of milestone research works on COVID-19 and low-quality preprints, which were eventually retracted, to verify whether the checklist can discriminate between the two categories. Results: At the external review step, 26 of the 54 contacted experts responded. The final checklist contained four elements (Research question, study type, transparency and integrity, and limitations), with ‘superficial’ and ‘deep’ levels for evaluation. When using both levels of evaluation, the checklist was effective at discriminating high- from low-quality preprints. Its usability was confirmed in workshops with our target audience: Bachelors students in Psychology and Medicine, and science journalists. Conclusions: We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind. We believe that our checklist has great potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this extends beyond COVID-19.

show abstract

Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Cited by 4 publications

References 31 publications

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Contact Info

Product

Resources

About