Is the future of peer review automated?

Abstract

The rising rate of preprints and publications, combined with persistent inadequate reporting practices and problems with study design and execution, have strained the traditional peer review system. Automated screening tools could potentially enhance peer review by helping authors, journal editors, and reviewers to identify beneficial practices and common problems in preprints or submitted manuscripts. Tools can screen many papers quickly, and may be particularly helpful in assessing compliance with journal policies and with straightforward items in reporting guidelines. However, existing tools cannot understand or interpret the paper in the context of the scientific literature. Tools cannot yet determine whether the methods used are suitable to answer the research question, or whether the data support the authors’ conclusions. Editors and peer reviewers are essential for assessing journal fit and the overall quality of a paper, including the experimental design, the soundness of the study’s conclusions, potential impact and innovation. Automated screening tools cannot replace peer review, but may aid authors, reviewers, and editors in improving scientific papers. Strategies for responsible use of automated tools in peer review may include setting performance criteria for tools, transparently reporting tool performance and use, and training users to interpret reports.

Meta‐analysis prediction intervals are under reported in sport and exercise medicine

“…An automated process could send a note to the journal editor when authors submit a random effects meta‐analysis without a prediction interval. Future research should focus on developing an automated prediction interval screening tool that could be implemented by journals, in similar way to other automated checks 32 …”

International multi‐stakeholder consensus statement on clinical trial integrity

“…This was specifically highlighted to be the case for the information related to reporting of ethics approval, sources of finding, potential conflict of interests, trial registration and statistical analysis plans 49–53 . In this regard, it is also foreseeable that journals in the future will develop and implement automated checks for RCT integrity just as they have done for the detection of plagiarism 54,55 …”

“…[49][50][51][52][53] In this regard, it is also foreseeable that journals in the future will develop and implement automated checks for RCT integrity just as they have done for the detection of plagiarism. 54,55 When an allegation of possible scientific misconduct is made, journals have an obligation to investigate in an unbiased manner with an explicit policy about managing conflicts of interests of their editors, peer-reviewers and advisors. Our statement advises authors to actively engage with the journal investigation process and submit their deidentifiable raw data to be examined if required.…”

Journal impact factors and the future of open access publishing

“…Automated tools may help journals streamline their review and publication processes. As described by Schulz et al., automated pre‐screening may help with improved reporting prior to review, or enhancing the peer review process 30 . They suggest that statistical errors, ethical lapses (such as plagiarism), and citations of retracted works are all easily identified with one such automated tool.…”