Towards automatic pulmonary nodule management in lung cancer screening with deep learning

Abstract

The introduction of lung cancer screening programs will produce an unprecedented amount of chest CT scans in the near future, which radiologists will have to read in order to decide on a patient follow-up strategy. According to the current guidelines, the workup of screen-detected nodules strongly relies on nodule size and nodule type. In this paper, we present a deep learning system based on multi-stream multi-scale convolutional networks, which automatically classifies all nodule types relevant for nodule workup. The system processes raw CT data containing a nodule without the need for any additional information such as nodule segmentation or nodule size and learns a representation of 3D data by analyzing an arbitrary number of 2D views of a given nodule. The deep learning system was trained with data from the Italian MILD screening trial and validated on an independent set of data from the Danish DLCST screening trial. We analyze the advantage of processing nodules at multiple scales with a multi-stream convolutional network architecture, and we show that the proposed deep learning system achieves performance at classifying nodule type that surpasses the one of classical machine learning approaches and is within the inter-observer variability among four experienced human observers.

A survey on deep learning in medical image analysis

“…As summarized in Table 2, most works employ simple Anavi et al (2015) Image retrieval Combines classical features with those from pre-trained CNN for image retrieval using SVM Bar et al (2015) Pathology detection Features from a pre-trained CNN and low level features are used to detect various diseases Anavi et al (2016) Image retrieval Continuation of Anavi et al (2015), adding age and gender as features Bar et al (2016) Pathology detection Continuation of Bar et al (2015), more experiments and adding feature selection Cicero et al (2016) Pathology detection GoogLeNet CNN detects five common abnormalities, trained and validated on a large data set Tuberculosis detection Processes entire radiographs with a pre-trained fine-tuned network with 6 convolution layers Kim and Hwang (2016) Tuberculosis detection MIL framework produces heat map of suspicious regions via deconvolution Shin et al (2016a) Pathology detection CNN detects 17 diseases, large data set (7k images), recurrent networks produce short captions Rajkomar et al (2017) Frontal/lateral classification Pre-trained CNN performs frontal/lateral classification task Yang et al (2016c) Bone suppression Cascade of CNNs at increasing resolution learns bone images from gradients of radiographs Wang et al (2016a) Nodule classification Combines classical features with CNN features from pre-trained ImageNet CNN Used a standard feature extractor and a pre-trained CNN to classify detected lesions as benign peri-fissural nodules van Detects nodules with pre-trained CNN features from orthogonal patches around candidate, classified with SVM Shen et al (2015b) Three CNNs at different scales estimate nodule malignancy scores of radiologists (LIDC-IDRI data set) Chen et al (2016e) Combines features from CNN, SDAE and classical features to characterize nodules from LIDC-IDRI data set Ciompi et al (2016) Multi-stream CNN to classify nodules into subtypes: solid, part-solid, non-solid, calcified, spiculated, perifissural Dou et al (2016b) Uses 3D CNN around nodule candidates; ranks #1 in LUNA16 nodule detection challenge Li et al (2016a) Detects nodules with 2D CNN that processes small patches around a nodule Setio et al (2016) Detects nodules with end-to-end trained multi-stream CNN with 9 patches per candidate Shen et al (2016) 3D CNN classifies volume centered on nodule as benign/malignant, results are combined to patient level prediction Sun et al (2016b) Same dataset as Shen et al (2015b)…”

Using Machine Learning to Identify Health Outcomes from Electronic Health Record Data

“…Machine learning could be useful for not only extracting details about a clinician’s impression from the diagnostic report (ie, as part of an NLP task), but also predicting the presence of the health outcome directly from features in the medical image itself (ie, as an image recognition task). In fact, an increasing number of studies are using deep learning models for a variety of medical-related image classification tasks, yielding impressive results (10, 43–46). For example, Rajpurkar et al (45) trained a 121-layer convolutional neural network called CheXNet to detect pneumonia from chest X-ray images.…”

Deep Learning for Medical Image Analysis: Applications to Computed Tomography and Magnetic Resonance Imaging

“…Ciompi et al proposed a single system that goes a step further than a CADe system by classifying nodules based on the morphology (solid, non-solid, part-solid, calcified, perifissural and spiculated) for automatic Lung-RADS reporting and malignancy estimation [16]. As seen in Fig 2, they performed data augmentation by extracting multiple view 2D patches from 3D nodule candidate voxels and used a multistream CNN architecture that fuses features extracted from multi-scale patches.…”