Data-centric artificial intelligence in oncology: a systematic review assessing data quality in machine learning models for head and neck cancer

Abstract

Machine learning models have been increasingly considered to model head and neck cancer outcomes for improved screening, diagnosis, treatment, and prognostication of the disease. As the concept of data-centric artificial intelligence is still incipient in healthcare systems, little is known about the data quality of the models proposed for clinical utility. This is important as it supports the generalizability of the models and data standardization. Therefore, this study overviews the quality of structured and unstructured data used for machine learning model construction in head and neck cancer. Relevant studies reporting on the use of machine learning models based on structured and unstructured custom datasets between January 2016 and June 2022 were sourced from PubMed, EMBASE, Scopus, and Web of Science electronic databases. Prediction model Risk of Bias Assessment (PROBAST) tool was used to assess the quality of individual studies before comprehensive data quality parameters were assessed according to the type of dataset used for model construction. A total of 159 studies were included in the review; 106 utilized structured datasets while 53 utilized unstructured datasets. Data quality assessments were deliberately performed for 14.2% of structured datasets and 11.3% of unstructured datasets before model construction. Class imbalance and data fairness were the most common limitations in data quality for both types of datasets while outlier detection and lack of representative outcome classes were common in structured and unstructured datasets respectively. Furthermore, this review found that class imbalance reduced the discriminatory performance for models based on structured datasets while higher image resolution and good class overlap resulted in better model performance using unstructured datasets during internal validation. Overall, data quality was infrequently assessed before the construction of ML models in head and neck cancer irrespective of the use of structured or unstructured datasets. To improve model generalizability, the assessments discussed in this study should be introduced during model construction to achieve data-centric intelligent systems for head and neck cancer management.

Head and neck cancer treatment outcome prediction: a comparison between machine learning with conventional radiomics features and deep learning radiomics

“…Due to the moderate number of included patients and the different primary tumor site distributions between the OUS and MAASTRO datasets ( Table 1 ), a mixed analysis, i.e., including all primary tumor sites in the datasets, was preferred over subgroup analysis, i.e., focusing on one single primary tumor site. Mixed analysis for HNC outcome prediction is also encountered frequently in the literature, as summarized in Adeoye et al ( 37 ).…”

Study Design and Research Protocol for diagnostic or prognostic studies in the Age of Artificial Intelligence: A Biostatistician’s Perspective

“…Noise in label assignment can originate from insufficient information, subjectivity, and coding issues. Regarding classes parity, a recent systematic review on data quality in AI models for head and neck cancer [19] suggested that models with good balance in the outcome classes had significantly higher median discrimination than those that did not adjust for classes imbalance.…”

Current State of Artificial Intelligence in Clinical Applications for Head and Neck MR Imaging

“…A review published in 2022 also indicated that among the AI-related studies focused on the head and neck, the number of investigations with the use of external validation in the test session is relatively small. 122 For an appropriate external-validation procedure, the use of large amounts of data from public academic institutions (e.g., J-MID from the Japan Radiology Society) may be one of the steps to solve this problem. 123 Further research is necessary to address these limitations, as are discussions to integrate the existing knowledge and future aspects.…”