Semi‐supervised training using cooperative labeling of weakly annotated data for nodule detection in chest CT

Abstract

Purpose: Machine learning algorithms are best trained with large quantities of accurately annotated samples. While natural scene images can often be labeled relatively cheaply and at large scale, obtaining accurate annotations for medical images is both time consuming and expensive. In this study, we propose a cooperative labeling method that allows us to make use of weakly annotated medical imaging data for the training of a machine learning algorithm. As most clinically produced data are weakly-annotated -produced for use by humans rather than machines and lacking information machine learning depends uponthis approach allows us to incorporate a wider range of clinical data and thereby increase the training set size. Methods: Our pseudo-labeling method consists of multiple stages. In the first stage, a previously established network is trained using a limited number of samples with high-quality expert-produced annotations. This network is used to generate annotations for a separate larger dataset that contains only weakly annotated scans. In the second stage, by cross-checking the two types of annotations against each other, we obtain higher-fidelity annotations. In the third stage, we extract training data from the weakly annotated scans, and combine it with the fully annotated data, producing a larger training dataset. We use this larger dataset to develop a computer-aided detection (CADe) system for nodule detection in chest CT. Results: We evaluated the proposed approach by presenting the network with different numbers of expert-annotated scans in training and then testing the CADe using an independent expert-annotated dataset. We demonstrate that when availability of expert annotations is severely limited, the inclusion of weakly-labeled data leads to a 5% improvement in the competitive performance metric (CPM), defined as the average of sensitivities at different false-positive rates. Conclusions: Our proposed approach can effectively merge a weaklyannotated dataset with a small, well-annotated dataset for algorithm training. This approach can help enlarge limited training data by leveraging the large amount of weakly labeled data typically generated in clinical image interpretation.

Regulatory considerations for medical imaging AI/ML devices in the United States: concepts and challenges

“…The AI/ML research program 80 strives to address these challenges by conducting peerreviewed research to develop and understand methods for enhanced AI/ML training, developing systematic approaches for understanding AI/ML robustness, and assessing novel test methodologies to evaluate fixed and continuously learning AI/ML performance in both the premarket and real-world settings, to name just a few areas of ongoing research. Some of the regulatory science projects being conducted as part of the OSEL AI/ML research program include a recent investigation developing a cooperative labeling technique to incorporate weakly labeled data into the training of a deep learning AI/ML model for lung nodule detection in CT. 81 This study showed that the inclusion of weakly labeled data leads to a 5% improvement in lung nodule detection performance when the number of expert annotations is limited. Another approach for addressing small dataset sizes is to augment available data with synthetic datasets.…”

“…Some of the regulatory science projects being conducted as part of the OSEL AI/ML research program include a recent investigation developing a cooperative labeling technique to incorporate weakly labeled data into the training of a deep learning AI/ML model for lung nodule detection in CT 81 . This study showed that the inclusion of weakly labeled data leads to a 5% improvement in lung nodule detection performance when the number of expert annotations is limited.…”

Training CADe algorithms with synthetic datasets: augmenting clinical data for improved lung nodule detection

“…In the first set of experiments, we train the false positive reduction algorithm 8 using augmentations of flip, rotation, and translation which are considered to be the baseline approach for common data augmentation methods for this application. 8 The result of these experiments are reported in Table 1. Each row of the table reports the sensitivity and CPM performance when a given number of LIDC scans are used for training, followed by the results when the LIDC dataset is complemented by the data from 569 phantom scans.…”

“…The training framework including the hyper-parameters and neural network architecture are selected in an identical manner to our previous false positive reduction network. 8 This network consists of successive threedimensional convolutions, max-poolings and Leaky ReLU activations, followed by two dense fully connected layers, assigning nodule scores to each ROI. The training data however is presented to the network with a new augmentation paradigm; during the training, each positive and negative ROI is passed through an image augmentation selected at random during the training.…”