Cardiothoracic ratio measurement using artificial intelligence: observer and method validation studies

Abstract

Background Artificial Intelligence (AI) is a promising tool for cardiothoracic ratio (CTR) measurement that has been technically validated but not clinically evaluated on a large dataset. We observed and validated AI and manual methods for CTR measurement using a large dataset and investigated the clinical utility of the AI method. Methods Five thousand normal chest x-rays and 2,517 images with cardiomegaly and CTR values, were analyzed using manual, AI-assisted, and AI-only methods. AI-only methods obtained CTR values from a VGG-16 U-Net model. An in-house software was used to aid the manual and AI-assisted measurements and to record operating time. Intra and inter-observer experiments were performed on manual and AI-assisted methods and the averages were used in a method variation study. AI outcomes were graded in the AI-assisted method as excellent (accepted by both users independently), good (required adjustment), and poor (failed outcome). Bland–Altman plot with coefficient of variation (CV), and coefficient of determination (R-squared) were used to evaluate agreement and correlation between measurements. Finally, the performance of a cardiomegaly classification test was evaluated using a CTR cutoff at the standard (0.5), optimum, and maximum sensitivity. Results Manual CTR measurements on cardiomegaly data were comparable to previous radiologist reports (CV of 2.13% vs 2.04%). The observer and method variations from the AI-only method were about three times higher than from the manual method (CV of 5.78% vs 2.13%). AI assistance resulted in 40% excellent, 56% good, and 4% poor grading. AI assistance significantly improved agreement on inter-observer measurement compared to manual methods (CV; bias: 1.72%; − 0.61% vs 2.13%; − 1.62%) and was faster to perform (2.2 ± 2.4 secs vs 10.6 ± 1.5 secs). The R-squared and classification-test were not reliable indicators to verify that the AI-only method could replace manual operation. Conclusions AI alone is not yet suitable to replace manual operations due to its high variation, but it is useful to assist the radiologist because it can reduce observer variation and operation time. Agreement of measurement should be used to compare AI and manual methods, rather than R-square or classification performance tests.

A clinical evaluation study of cardiothoracic ratio measurement using artificial intelligence

“…The preferred result is when the user can accept the AI results without further interaction, which is considered an excellent grade result in our study. In our 2021 study [ 9 ] of the AI-assisted method, we found that our model could achieve an excellent grade in only about 40% of images, lower than our desired result of around 70%. In a more recent study [ 10 ], we developed an improved model architecture and better training methodology that achieved CTR measurement with an average error on-par with manual measurement by experts.…”

“…Due to the retrospective nature of the study, informed consent was not required. The validation dataset was from our previous investigation (Si069/2020) of observer and method validation [ 9 ], and was employed here to compare the performance of our improved DL models to the previous one. Briefly, there were 7517 PA-upright-CXR images acquired between 2010 and 2019 from patients >17 years of age, from randomly selected normal images (5000) and all cardiomegaly images with CTR measurement reports (2517).…”

“…In comparison with the model used in our previous study [ 9 ], this model set was vastly improved by (1) adding new model architecture and performing hyper-parameter optimization, (2) expanding our segmented training dataset, and (3) expanding our image augmentation repertoire to improve generalizability.…”

“…To ease the burden, Deep Learning (DL), a subset of Artificial Intelligence (AI), has been implemented for CTR calculation [ 5 – 11 ]. The AI method had been technically [ 6 – 8 ] and clinically [ 9 , 10 ] validated for CTR measurement and can provide a reliable result with measurement variation within the human-operating range [ 10 ]. Such reliability made the automated calculation of the CTR feasible, but in actual clinical practice automated measurement has not been employed [ 9 ] because the measurements still required approval from radiologists.…”

“…Here, we further investigated the efficiency and reliability of all models from our recent study [ 10 ] using the AI-assisted method, and aimed to find the best model for clinical use. We performed a validation study on the models using our previous dataset [ 9 ] with manual calculation of the CTR measurement as the reference, and compared the performance of these models to find the best option for clinical implementation (i.e., the model that provided the highest proportion of excellent grade results). We then evaluated the selected model on evaluation dataset for clinical use to determine the model’s efficiency to assist radiologists to measure the CTR on all normal and cardiomegaly cases.…”