Best practices for authors of healthcare-related artificial intelligence manuscripts

Abstract

Since its inception in 2017, npj Digital Medicine has attracted a disproportionate number of manuscripts reporting on uses of artificial intelligence. This field has matured rapidly in the past several years. There was initial fascination with the algorithms themselves (machine learning, deep learning, convoluted neural networks) and the use of these algorithms to make predictions that often surpassed prevailing benchmarks. As the discipline has matured, individuals have called attention to aberrancies in the output of these algorithms. In particular, criticisms have been widely circulated that algorithmically developed models may have limited generalizability due to overfitting to the training data and may systematically perpetuate various forms of biases inherent in the training data, including race, gender, age, and health state or fitness level (Challen et al. BMJ Qual. Saf. 28:231–237, 2019; O’neil. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Broadway Book, 2016). Given our interest in publishing the highest quality papers and the growing volume of submissions using AI algorithms, we offer a list of criteria that authors should consider before submitting papers to npj Digital Medicine.

Preventing dataset shift from breaking machine-learning biomarkers

“…To obtain valid predictive models that perform well beyond the training sample, it is crucial to collect datasets that represent the whole population and reflect its diversity as much as possible [ 6 , 25 , 26 ]. Yet clinical research often emphasizes the opposite: very homogeneous datasets and carefully selected participants.…”

“…One risk is the potential mismatch, or “dataset shift," between the distribution of the individuals used to estimate this statistical link and that of the target population that should benefit from the biomarker. In this case, the extracted associations may not apply to the target population [ 6 ]. Computer-aided diagnostics of thoracic diseases from X-ray images has indeed been shown to be unreliable for individuals of a given sex if built from a cohort over-representing the other sex [ 7 ].…”

EVIDENCE Publication Checklist for Studies Evaluating Connected Sensor Technologies: Explanation and Elaboration

“…All methods used to perform analytical and clinical validation should be included in the statistical analysis section. Further details on good practices can be seen here [41]. To increase transparency and enable reproducibility, authors are encouraged to share their work on public code repositories, if applicable [23,[42][43][44].…”

Artificial intelligence for the next generation of precision oncology

“…There have been several initiatives proposing "best practice" guidelines to ensure that AI methods are developed and deployed in a way that maximizes the benefit for patients. On the model development front, these include recommendations for providing sufficient methodological detail on the development of algorithms, and encouraging sharing of data sets and code to enhance the transparency in the reporting of AI algorithms in medicine 11,12,24,25 and allow other researchers to determine the rigor, quality, reproducibility, and generalizability of the findings [26][27][28] . Studies also need to adopt standardized guidelines for describing and reporting aspects related to the purpose and context of the "clinical need" that is being addressed, the quality of data used to train the models-including issues related to power calculation, labeling, model biases, etc.-measures of performance, outputs and framework for integration in clinical pathways and workflows, among others.…”