Limited generalizability of deep learning algorithm for pediatric pneumonia classification on external data

Abstract

Purpose (1) Develop a deep learning system (DLS) to identify pneumonia in pediatric chest radiographs, and (2) evaluate its generalizability by comparing its performance on internal versus external test datasets. Methods Radiographs of patients between 1 and 5 years old from the Guangzhou Women and Children’s Medical Center (Guangzhou dataset) and NIH ChestXray14 dataset were included. We utilized 5232 radiographs from the Guangzhou dataset to train a ResNet-50 deep convolutional neural network (DCNN) to identify pediatric pneumonia. DCNN testing was performed on a holdout set of 624 radiographs from the Guangzhou dataset (internal test set) and 383 radiographs from the NIH ChestXray14 dataset (external test set). Receiver operating characteristic curves were generated, and area under the curve (AUC) was compared via DeLong parametric method. Colored heatmaps were generated using class activation mapping (CAM) to identify important image pixels for DCNN decision-making. Results The DCNN achieved AUC of 0.95 and 0.54 for identifying pneumonia on internal and external test sets, respectively (p < 0.0001). Heatmaps generated by the DCNN showed the algorithm focused on clinically relevant features for images from the internal test set, but not for images from the external test set. Conclusion Our model had high performance when tested on an internal dataset but significantly lower accuracy when tested on an external dataset. Likewise, marked differences existed in the clinical relevance of features highlighted by heatmaps generated from internal versus external datasets. This study underscores potential limitations in the generalizability of such DLS models.

A review of PET attenuation correction methods for PET-MR

“…One of the limiting factors in the majority of published studies is the lack of testing on external datasets with the validation data being used instead for assessing the performance of the method [ 273 ]. This is a general issue in the field of AI and deep learning that could lead to “data leakage”.…”

Improving the Generalizability and Performance of an Ultrasound Deep Learning Model Using Limited Multicenter Data for Lung Sliding Artifact Identification

“…The challenges of obtaining and working with high-quality, external imaging data often stunt models from gaining the necessary validation required for eventual clinical deployment. Furthermore, even when external data are available, performance degradation is routinely observed when models are tested against external datasets [ 6 , 7 , 8 , 9 ]. Point-of-care ultrasound data presents additional unique challenges including various manufacturers, scanning presets, and probe types used based on individual institutional practices.…”

Uncovering the effects of model initialization on deep model generalization: A study with adult and pediatric chest X-ray images

“…The significance of model initialization is amplified when we consider challenges in model generalization which are primarily due to feature distribution shifts between training datasets and real-world use. For example, a model trained and tested on adult CXR data from the same source (internal testing) may result in significantly higher performance compared to testing it on adult CXR data from another source (external testing) [11]. Additional performance degradation may be observed when pediatric images exhibiting the same disease(s) are included in the testing.…”

“…Model generalizability is defined as the ability of a trained model to capture generalized patterns and perform well on unseen data [18]. Medical computer vision relies on model generalizability for several reasons [11] including accommodating patient diversity, adapting to various data sources and quality, addressing ethical considerations, and enhancing clinical utility. A general model is robust to different data sources and population distributions, considering factors such as the patient/study subject's ethnicity, sex, and severity of the disease(s) expressed on the image.…”

“…These discrepancies present challenges for DL models trained on adult data when directly applied to pediatric cases, potentially leading to sub-optimal generalizability, and reduced clinical utility. Prior work in pediatric CXR image analysis includes the development and evaluation of a ResNet-50 model trained to classify pediatric CXRs as showing pneumonia-consistent manifestations or normal lungs [11]. The model demonstrated comparatively improved performance on the internal test set (area under the curve (AUC): 0.95) compared to the external NIH-CXR test set (AUC: 0.54), highlighting potential limitations in model generalizability.…”