Deep semantic segmentation of natural and medical images: a review

Taghanaki, Saeid Asgari; Abhishek, Kumar; Cohen, Joseph; Cohen‐Adad, Julien; Hamarneh, Ghassan

doi:10.1007/s10462-020-09854-1

Cited by 614 publications

(312 citation statements)

References 183 publications

Supporting

Mentioning

311

Contrasting

Unclassified

Order By: Relevance

“…The softmax operation is performed to ensure that the prediction result is finally mapped into the (0,1) interval, which is used to represent the probability that the pixels are the background or the disc. As the most commonly used loss function, cross-entropy loss examines each pixel independently and compares the class prediction vector with ground-truth [ 15 ]. Then, cross entropy (CE) can be defined as:

where

is the groundtruth class, and

[0, 1] is the prediction class.…”

Section: Methodsmentioning

confidence: 99%

“…Attention mechanism is gradually gaining popularity in medical segmentation. The attention mechanism can be viewed as using feature map information to select and locate the most significant part of the input signal [ 15 ]. Hu et al [ 16 ] used global average pooling to aggregate feature map information, then reduced it to a single channel feature map, and finally used an activation gate to highlight salient features.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Optic Disc Segmentation Using Attention-Based U-Net and the Improved Cross-Entropy Convolutional Neural Network

Jin

Wang

et al. 2020

Entropy

View full text Add to dashboard Cite

Medical image segmentation is an important part of medical image analysis. With the rapid development of convolutional neural networks in image processing, deep learning methods have achieved great success in the field of medical image processing. Deep learning is also used in the field of auxiliary diagnosis of glaucoma, and the effective segmentation of the optic disc area plays an important assistant role in the diagnosis of doctors in the clinical diagnosis of glaucoma. Previously, many U-Net-based optic disc segmentation methods have been proposed. However, the channel dependence of different levels of features is ignored. The performance of fundus image segmentation in small areas is not satisfactory. In this paper, we propose a new aggregation channel attention network to make full use of the influence of context information on semantic segmentation. Different from the existing attention mechanism, we exploit channel dependencies and integrate information of different scales into the attention mechanism. At the same time, we improved the basic classification framework based on cross entropy, combined the dice coefficient and cross entropy, and balanced the contribution of dice coefficients and cross entropy loss to the segmentation task, which enhanced the performance of the network in small area segmentation. The network retains more image features, restores the significant features more accurately, and further improves the segmentation performance of medical images. We apply it to the fundus optic disc segmentation task. We demonstrate the segmentation performance of the model on the Messidor dataset and the RIM-ONE dataset, and evaluate the proposed architecture. Experimental results show that our network architecture improves the prediction performance of the base architectures under different datasets while maintaining the computational efficiency. The results render that the proposed technologies improve the segmentation with 0.0469 overlapping error on Messidor.

show abstract

where

is the groundtruth class, and

[0, 1] is the prediction class.…”

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Optic Disc Segmentation Using Attention-Based U-Net and the Improved Cross-Entropy Convolutional Neural Network

Jin

Wang

et al. 2020

Entropy

View full text Add to dashboard Cite

show abstract

“…In recent years, deep learning has developed rapidly in the field of computer vision. It has made great progress in image classification [12][13][14][15][16][17][18], object detection [19,20] and image segmentation [22][23][24][25][26][27]. Compared with traditional methods, deep neural networks can automatically extract features from the input data and achieve higher accuracy.…”

Section: Related Workmentioning

confidence: 99%

A Novel Adaptive Weighted Loss Design in Adversarial Learning for Retinal Nerve Fiber Layer Defect Segmentation

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Glaucoma is a chronic eye disease that can cause permanent visual loss and is difficult to detect early. Retinal nerve fiber layer defect (RNFLD) is clinical evidence for the diagnosis of glaucoma. Classical deep learning based methods can be used to segment RNFLD from fundus images. However, the segmentation results of these methods do not have the specific geometry of RNFLD, and the segmentation errors of fundus images with special styles are large. In this paper, we present a novel conditional adversarial shuffle U-shaped network (CASU-Net) to segment RNFLD, which consists of a generator and a discriminator. For the generator, a mixed loss is designed, which consists of an adaptive weighted segmentation loss and an adversarial loss. This adaptive weighted segmentation loss can balance the segmentation accuracy of the target and background region, and assign more attention to the hard samples, thus ensuring the consistent improvement of the segmentation accuracy of all fundus images. The adversarial loss not only helps to improve the pixel-wise segmentation accuracy but also makes the geometry of the RNFLD segmentation closer to the ground truth. In addition, in the generator, a shuffle module was designed to fully mine the information of all channels to improve the feature extraction capability of the model. The proposed CASU-Net is verified on a RNFLD dataset from Beijing Tongren Hospital. The experiments show that the CASU-Net achieves state-of-the-art results on this dataset. INDEX TERMS Glaucoma, retinal nerve fiber layer defect segmentation, deep learning.

show abstract

“…Compared to using a single modality, multi-modalities significantly improve the performance of learning models [13,14,15,16,17]. Several relevant surveys already exist, such as deep learning-based semantic segmentation [2,3,18,19], indoor scene understanding [20,21], multimodal perception for autonomous driving [22], multimodal human motion recognition [23], multimodal medical image segmentation [24], and multimodal learning study [25,26]. However, these review works are mostly focused on unimodal image segmentation, multimodal fusion for specific domains, or multimedia analysis across video, audio, and text.…”

Section: Introductionmentioning

confidence: 99%

Deep multimodal fusion for semantic image segmentation: A survey

Zhang

Sidibé

Morel

et al. 2021

Image and Vision Computing

144

View full text Add to dashboard Cite

Recent advances in deep learning have shown excellent performance in various scene understanding tasks. However, in some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies have demonstrated that deep multimodal fusion for semantic image segmentation achieves significant performance improvement. These fusion approaches take the benefits of multiple information sources and generate an optimal joint prediction automatically. This paper describes the essential background concepts of deep multimodal fusion and the relevant applications in computer vision. In particular, we provide a systematic survey of multimodal fusion methodologies, multimodal segmentation datasets, and quantitative evaluations on the benchmark datasets. Existing fusion methods are summarized according to a common taxonomy: early fusion, late fusion, and hybrid fusion. Based on their performance, we analyze the strengths and weaknesses of different fusion strategies. Current challenges and design choices are discussed, aiming to provide the reader with a comprehensive and heuristic view of deep multimodal image segmentation.

show abstract

Deep semantic segmentation of natural and medical images: a review

Cited by 614 publications

References 183 publications

Optic Disc Segmentation Using Attention-Based U-Net and the Improved Cross-Entropy Convolutional Neural Network

Optic Disc Segmentation Using Attention-Based U-Net and the Improved Cross-Entropy Convolutional Neural Network

A Novel Adaptive Weighted Loss Design in Adversarial Learning for Retinal Nerve Fiber Layer Defect Segmentation

Deep multimodal fusion for semantic image segmentation: A survey

Contact Info

Product

Resources

About