External validation of the performance of commercially available deep-learning-based lung nodule detection on low-dose CT images for lung cancer screening in Japan

Abstract

Purpose Artificial intelligence (AI) algorithms for lung nodule detection have been developed to assist radiologists. However, external validation of its performance on low-dose CT (LDCT) images is insufficient. We examined the performance of the commercially available deep-learning-based lung nodule detection (DL-LND) using LDCT images at Japanese lung cancer screening (LCS). Materials and methods Included were 43 patients with suspected lung cancer on LDCT images and pathologically confirmed lung cancer. The reference standard for nodules whose diameter exceeded 4 mm was set by a radiologist who referred to the reports of two other radiologists reading the LDCT images. After we applied commercially available DL-LND to the LDCT images, the radiologist reviewed all nodules detected by DL-LND. When he failed to identify an existing nodule, it was also included in the reference standard. To validate the performance of DL-LND, the sensitivity for lung nodules and lung cancer, the positive-predictive value (PPV) for lung nodules, and the mean number of false-positive (FP) nodules per CT scan were recorded. Results The radiologist detected 97 nodules including 43 lung cancers and missed 3 solid nodules detected by DL-LND. A total of 100 nodules was included in the reference standard. DL-LND detected 396 nodules including 40 lung cancers. The sensitivity for the 100 nodules was 96.0%; the PPV was 24.2% (96/396). The mean number of FP nodules per CT scan was 7.0; sensitivity for lung cancer was 93.0% (40/43). DL-LND missed three lung cancers; 2 of these were atypical pulmonary cysts. Conclusion We externally verified that the sensitivity for lung nodules and lung cancer by DL-LND was very high. However, its low PPV and the increased FP nodules remains a serious drawback of DL-LND.