External Validation of Deep Learning Algorithms for Cardiothoracic Ratio Measurement

Abstract

Recent advances in machine learning have made it possible to create automated systems for medical image diagnosis. Cardiothoracic ratio (CTR) measurement, a common procedure for assessing cardiac abnormality in chest radiographs, has been investigated by several deep learning studies aiming to automate the process. However, of key consideration is whether automated CTR measurements by machine learning models can yield CTR values as accurately and consistently as trained human technicians on unseen data and thereby be considered trustworthy for clinical application. To assess this, we performed external validation of automated CTR algorithms on a dataset of 7,517 images, comparing four variants of U-Net architecture in heart and lung segmentations and CTR calculations: VGG-11, VGG-16, SegNet, and AlbuNet. We then benchmarked their performance against two human experts manually measuring CTR on the same images in a clinical setting such that we could equitably compare model-to-human variation against human-to-human variation. Our analysis shows that AlbuNet demonstrates human-level performance in CTR measurements, achieving MAPE of 2.38%, which is on par with human-to-human inter-rater variability (2.53%) when using the manual measurement method. The other three U-Net variants, particularly VGG-16, also performed similarly well. Additionally, we conducted an extreme outlier analysis on each model architecture, assessing the percentage of samples with higher measurement errors than the maximum error from the manual method. AlbuNet outperformed other architectures with only 0.35% occurrence of extreme outliers, while the other three U-Net variants ranged from 0.64% to 1.06% occurrence. Overall, the deep-learning-based algorithm was demonstrated to be as reliable as the manual method and shows strong potential for assisting radiologists in the CTR measurement process in clinical practice.

A clinical evaluation study of cardiothoracic ratio measurement using artificial intelligence

“…In our recent study [ 10 ], we reviewed the literature regarding anatomical segmentation in chest x-rays and observed that U-Net has emerged as a widely used model for chest x-ray and medical image segmentation tasks [ 12 , 13 ]. As the name suggested, the U-shape architecture consists of (1) an encoder that extracts features through successive convolutional layers that reduce the dimension of the inputs, and (2) a decoder that applies successive up-sampling operators to predict a high-resolution mask output.…”

“…First, we validated the proposed DL models [ 10 ] to find the best model results for clinical implementation, and then evaluated the best model for clinical use. To validate the DL models, we performed the experiment on our previous dataset with manual results that served as the reference and employed the models using the AI-assisted method [ 9 ], and calculated percentage difference of CTR values between AI’s and manual results, or CTR diff .…”

“…To ease the burden, Deep Learning (DL), a subset of Artificial Intelligence (AI), has been implemented for CTR calculation [ 5 – 11 ]. The AI method had been technically [ 6 – 8 ] and clinically [ 9 , 10 ] validated for CTR measurement and can provide a reliable result with measurement variation within the human-operating range [ 10 ]. Such reliability made the automated calculation of the CTR feasible, but in actual clinical practice automated measurement has not been employed [ 9 ] because the measurements still required approval from radiologists.…”

“…In our 2021 study [ 9 ] of the AI-assisted method, we found that our model could achieve an excellent grade in only about 40% of images, lower than our desired result of around 70%. In a more recent study [ 10 ], we developed an improved model architecture and better training methodology that achieved CTR measurement with an average error on-par with manual measurement by experts. The study concluded that the improved AlbuNet model could be reliably employed for the automated calculation of CTR values.…”

“…Here, we further investigated the efficiency and reliability of all models from our recent study [ 10 ] using the AI-assisted method, and aimed to find the best model for clinical use. We performed a validation study on the models using our previous dataset [ 9 ] with manual calculation of the CTR measurement as the reference, and compared the performance of these models to find the best option for clinical implementation (i.e., the model that provided the highest proportion of excellent grade results).…”