DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks

Abstract

In this paper, we propose DeepCut, a method to obtain pixelwise object segmentations given an image dataset labelled weak annotations, in our case bounding boxes. It extends the approach of the well-known GrabCut [1] method to include machine learning by training a neural network classifier from bounding box annotations. We formulate the problem as an energy minimisation problem over a densely-connected conditional random field and iteratively update the training targets to obtain pixelwise object segmentations. Additionally, we propose variants of the DeepCut method and compare those to a naïve approach to CNN training under weak supervision. We test its applicability to solve brain and lung segmentation problems on a challenging fetal magnetic resonance dataset and obtain encouraging results in terms of accuracy.

Cardiac substructure segmentation with deep learning for improved cardiac sparing

“…Aside from removing spurious outlying points, CRFs also improved the smoothed appearance of the segmentations as needed for clinical application. 55 CRF tuning required different parameters for cardiac substructures based on size and shape, much like the work completed by Rajchl et al 56 The improvement in segmentation agreement observed, along with the use of a 3D-CRF to remove spurious isolated regions, parallels other emerging uses of 3D-CRF post-processing in medical imaging. 57,58 Although this study implemented CRFs as a post-processing step, some current studies have integrated CRFs into the utilized neural network and have seen improved segmentation performance 42,43,59 and can be explored in future work for possible coronary artery segmentation improvement.…”

“…Aside from removing spurious outlying points, CRFs also improved the smoothed appearance of the segmentations as needed for clinical application . CRF tuning required different parameters for cardiac substructures based on size and shape, much like the work completed by Rajchl et al . The improvement in segmentation agreement observed, along with the use of a 3D‐CRF to remove spurious isolated regions, parallels other emerging uses of 3D‐CRF post‐processing in medical imaging .…”

A survey on deep learning in medical image analysis

“…Several researchers have shown that designing architectures incorporating unique task-specific properties can obtain better results than straightforward CNNs. Two examples which we encountered several times are multi-view Gao et al (2016d) Frame labeling US CNN 4 class frame classification using transfer learning with pre-trained networks Kumar et al (2016) Frame labeling US CNN 12 standard anatomical planes, CNN extracts features for support vector machine Rajchl et al (2016b) Segmentation with non expert labels MRI CNN Crowd-sourcing annotation efforts to segment brain structures Rajchl et al (2016a) Segmentation given bounding box MRI CNN CNN and CRF for segmentation of structures Ravishankar et al (2016a) Quantification US CNN Hybrid system using CNN and texture features to find abdominal circumference Yu et al (2016b) Left ventricle segmentation US CNN Frame-by-frame segmentation by dynamically fine-tuning CNN to the latest frame Wound segmentation photographs CNN Additional detection of infection risk and healing progress Ypsilantis et al (2015) Chemotherapy response prediction PET CNN CNN outperforms classical radiomics features in patients with esophageal cancer Zheng et al (2015) Carotid artery bifurcation detection CT CNN Two stage detection process, CNNs combined with Haar features Alansary et al (2016) Placenta segmentation MRI CNN 3D multi-stream CNN with extension for motion correction Fritscher et al (2016) Head&Neck tumor segmentation CT CNN 3 orthogonal patches in 2D CNNs, combined with other features Jaumard- Hakoun et al (2016) Tongue contour extraction US RBM Analysis of tongue motion during speech, combines auto-encoders with RBMs Payer et al (2016) Hand landmark detection X-ray CNN Various architectures are compared Quinn et al (2016) Disease detection microscopy CNN Smartphone mounted on microscope detects malaria, tuberculosis & parasite eggs Smistad and Løvstakken (2016) Vessel detection and segmentation US CNN Femoral and carotid vessels analyzed with standard fCNN Twinanda et al (2017) Task recognition in laparoscopy Videos CNN Fine-tuned AlexNet applied to video frames Xu et al (2016c) Cervical dysplasia cervigrams CNN Fine-tuned pre-trained network with added non-imaging features Xue et al (2016) Esophageal microvessel classification Microscopy CNN Simple CNN used for feature extraction Zhang et al (2016a) Image reconstruction CT CNN Reconstructing from limited angle measurements, reducing reconstruction artefacts Lekadir et al (2017) Carotid plaque classification US CNN Simple CNN for characterization of carotid plaque composition in ultrasound …”

Adversarial Deep Structural Networks for Mammographic Mass Segmentation

“…Liskowski and Krawiec trained deep convolutional network on a large dataset to do retinal segmentation, and obtained impressive results [18]. The DeepCut network iteratively updated weak annotations and learned the convolutional neural network followed by CRF to do brain and lung segmentation in MR images [26].…”