The Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review (Preprint)

Abstract

BACKGROUND Pediatric diabetes is among the leading areas that adopt digital technologies and intelligent devices. However, there has been growing concern about the transparency, replicabiliby, biasedness and overall validity of research in the field of medical artificial intelligence and machine learning. OBJECTIVE To systematically review the reporting quality of machine learning (ML) studies on pediatric diabetes mellitus. METHODS PubMed and Web of Science databases were searched for 2016-2020. Studies were included if use of ML was reported in 2–18-year-old children with diabetes mellitus, including studies of complications, screening studies and in silico samples. In studies following the ML workflow of training, validation, and evaluation of results, reporting quality was assessed via the Minimum Information about CLinical Artificial Intelligence Modelling (MI-CLAIM) checklist by pairs of independent reviewers. Positive answers to the 17 binary items about sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. Synthesis of results included correlation analysis of reporting quality with date of publication, and testing differences by data type, subjects (human or in silico), research goals, level of code sharing, and the scientific field of publications (medial or engineering) was evaluated via ANOVA or t-tests. The association of MI-CLAIM ratings with expert judgements of clinical impact and reproducibility was also tested. RESULTS After screening 1043 records 28 studies were included. The sample size of training cohorts ranged between 5-561. Six studies featured only in silico patients. 61 unique ML techniques were applied. Reporting quality was low with great variation among the 21 assessed studies. The number of items with sufficient reporting ranged between 4-12 (mean 7.43). The research questions and data characterization were most often, while patient characteristics and model examination were least often reported adequately. Reporting quality improved over time (r=0.50, P=.02), it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in non-invasive hypoglycemia detection studies (P=.006), higher in studies published in medical vs. engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline vs. no sharing (P=.003). Studies with human or in silico samples or using various data types did not differ in reporting quality. The association between expert judgements and MI-CLAIM ratings was not significant. CONCLUSIONS The reporting quality of ML studies in the pediatric population with diabetes was generally low. Important details for clinicians, such as patient characteristics, comparison to the state-of-the-art solution and model examination for valid, unbiased, and robust results were often the weak points of reporting. To allow the assessment of their clinical utility, the reporting standards of machine learning studies must evolve, and algorithms for this challenging population must become more transparent and replicable.