Background Machine learning has been an emerging tool for various aspects of infectious diseases including tuberculosis surveillance and detection. However, the World Health Organization (WHO) provided no recommendations on using computer-aided tuberculosis detection software because of a small number of studies, methodological limitations, and limited generalizability of the findings. Methods To quantify the generalizability of the machine-learning model, we developed a Deep Convolutional Neural Network (DCNN) model using a Tuberculosis (TB)-specific chest x-ray (CXR) dataset of one population (National Library of Medicine Shenzhen No.3 Hospital) and tested it with non-TB-specific CXR dataset of another population (National Institute of Health Clinical Centers). Results In the training and intramural test sets using the Shenzhen hospital database, the DCCN model exhibited an AUC of 0.9845 and 0.8502 for detecting TB, respectively. However, the AUC of the supervised DCNN model in the ChestX-ray8 dataset was dramatically dropped to 0.7054. Using the cut points at 0.90, which suggested 72% sensitivity and 82% specificity in the Shenzhen dataset, the final DCNN model estimated that 36.51% of abnormal radiographs in the ChestX-ray8 dataset were related to TB. Conclusion A supervised deep learning model developed by using the training dataset from one population may not have the same diagnostic performance in another population. Conclusion: Technical specification of CXR images, disease severity distribution, dataset distribution shift, and overdiagnosis should be examined before implementation in other settings.

BackgroundAutomated classification of chest radiograph (CXR) using deep convolutional neural network (DCCN) has emerged as an attractive option for tuberculosis surveillance and detection. The National Institute of Health (NIH) ChestX-ray8 database comprises 32,717 patients with X-ray images that were interpreted as abnormal based on natural language processing.MethodsTwo de-identified HIPAA-compliant datasets including the NIH ChestX-ray8 database and the National Library of Medicine (NLM) Shenzhen Hospital X-ray set were included in this study. First, Shenzhen Hospital X-ray set which consisted of 336 chest radiographs related to TB and 326 normal radiographs were used to develop DCCN. The dataset was split into training (75%), validation (15%), and test (10%). Based on TensorFlow framework, Inception-v3, a novel pre-trained DCCN, was augmented with several techniques to classify an image as having TB characteristics or as healthy. Receiver operating characteristic (ROC) curves and areas under the curve (AUCs) were used to assess model performance. Next, 89,845 radiographs (38,086 normal and 51,759 abnormal images) from ChestX-ray8 dataset which comprises 63,061 normal radiographs and 51,759 chest radiographs with one of eight common thoracic abnormality (including atelectasis, cardiomegaly, effusion, infiltration, mass, nodule,pneumonia and pneumothorax) were used to create the second test set to assess external validity of DCCN model. At the end, prevalence of tuberculosis associated radiographs in the US radiograph dataset was calculated by using trained DCCN model.ResultsThe final DCCN model had an AUCs of 0.96 to classify TB associated and normal chest radiographs in Shenzhen Hospital X-ray set. However, AUCs of trained DCCN to classify abnormal radiographs in ChestX-ray8 dataset was decreased to 0.54. Lastly, the final DCCN model predicted that there was 36.51% (13,905 of 51,759) of abnormal radiographs in ChestX-ray8 dataset related to tuberculosis.ConclusionOur trained DCCN model suggested 36.51% of abnormal chest radiography in the US dataset was associated with TB. However, AUCs of DCCN to classify normal chest radiograph differed upon settings and source of training set. Further researches should focus on improving efficacy of deep learning algorithm across various databases.Disclosures All authors: No reported disclosures.