Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond

Abstract

Background: The quality of COVID-19 preprints should be considered with great care, as we have seen that their contents can influence public policy. Efforts to improve preprint quality have mostly focused on introducing quick peer review, but surprisingly little has been done to calibrate the public’s evaluation of preprints and their contents. The aim of the PRECHECK project was to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and in general.Methods: To create a checklist, we applied a 4-step procedure consisting of an initial internal review, followed by an external review by a pool of experts methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and an implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. Additionally, after each internal review round, we applied the checklist on a set of high-quality preprints (i.e., from an online list of milestone research works in the COVID-19 pandemic) and low-quality preprints (i.e., that were eventually retracted) to verify whether the checklist can discriminate between the two categories.Results: At the external review step, of 54 experts contacted, 26 responded. The final checklist contained 4 elements (Research question, Study type, Transparency and integrity, and Limitations), with ‘superficial’ and ‘deep’ levels at which preprints can be evaluated. When using both levels of evaluation, the checklist was effective at discriminating high- from low-quality preprints. Its usability was confirmed in workshops with members of our target audience: Bachelors students in Psychology and Medicine, as well as with science journalists.Conclusions: We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind. We thus believe that our checklist has great potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this may very well extend beyond COVID-19.

The publication facts label: A public and professional guide for research articles

“…In that regard, at least one study has shown that using integrity ratings, gauges, and checkmarks (as well as red exclamation points) proved less useful than simply presenting the underlying data (Løvlie et al, 2023). Nor does the PFL rely on a close reading of the study by knowledgeable readers, although for such readers, a research evaluation checklist has proven effective with preprints (Turoman et al, 2023). The PRL provides just the facts, including the authors' competing interest statements and a link to board‐member profiles.…”