CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Abstract

Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. 1

An adversarial approach for the robust classification of pneumonia from chest radiographs

“…We note, however, that pneumonia is a diagnosis that is made in the context of clinical evidence of disease, and a disease where there is not necessarily perfect concordance between severity of symptoms and radiographic evidence of infiltrate [3,20,21,36]. In the description of the creation of the "Pneumonia" label in the CheXpert dataset, the authors note that while pneumonia is a clinical diagnosis, "Pneumonia... was included as a label in order to represent the images that suggested primary infection as the diagnosis, " suggesting that clinical information may play a role in labeling [13]. Disentangling the relationship between radiographic evidence of consolidation, the clinical presence of pneumonia symptoms, and the influence of the latter on the labeling of the former in these datasets could be helpful.…”

“…To assess the robustness of models to dataset shift, we used chest radiographs from two large publicly-available datasets. For our model training source domain, we used the CheXpert dataset from Stanford [13]. This dataset contains 224,316 chest radiographs of 65,240 patients.…”

Breaking Medical Data Sharing Boundaries by Employing Artificial Radiographs

“…The second dataset used in this study is the CheXpert dataset, which has been released by Irvin et al in January 2019 and contains 224,316 chest radiographs of 65,240 patients [39]. This dataset was used to train a second GAN in order to demonstrate the feasibility of the proposed data sharing approach (see Figure 1).…”

“…2. According to the labeling performance comparison mentioned in [39], most of the uncertainty labels (U) were assigned to 1.0, except for the consolidation class.…”

“…The seven overlapping labels between the NIH and Stanford dataset are listed in the table above. Note that pathological radiographs seem to be more common in the Stanford CheXpert dataset due to the differences in the labelling process [39]. Abbreviations: NIH, National Institutes of Health Clinical Center; SD, standard deviation.…”