Exploring New Backbone and Attention Module for Semantic Segmentation in Street Scenes

Fan, Lei; Wang, Wei-Chien; Zha, F.S.; Yan, Jiapeng

doi:10.1109/access.2018.2880877

Cited by 21 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Frequency Weighted Intersection over Union Besides, many elaborated backbone networks, such as VGGNet [81], ResNet [82], and Xception [172], are widely used in a variety of segmentation network design [173]. These backbone networks not only extract effective semantic information and spatial details but also simplify the training.…”

Section: Discussionmentioning

confidence: 99%

Deep multimodal fusion for semantic image segmentation: A survey

Zhang

Sidibé

Morel

et al. 2021

Image and Vision Computing

145

View full text Add to dashboard Cite

Recent advances in deep learning have shown excellent performance in various scene understanding tasks. However, in some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies have demonstrated that deep multimodal fusion for semantic image segmentation achieves significant performance improvement. These fusion approaches take the benefits of multiple information sources and generate an optimal joint prediction automatically. This paper describes the essential background concepts of deep multimodal fusion and the relevant applications in computer vision. In particular, we provide a systematic survey of multimodal fusion methodologies, multimodal segmentation datasets, and quantitative evaluations on the benchmark datasets. Existing fusion methods are summarized according to a common taxonomy: early fusion, late fusion, and hybrid fusion. Based on their performance, we analyze the strengths and weaknesses of different fusion strategies. Current challenges and design choices are discussed, aiming to provide the reader with a comprehensive and heuristic view of deep multimodal image segmentation.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep multimodal fusion for semantic image segmentation: A survey

Zhang

Sidibé

Morel

et al. 2021

Image and Vision Computing

145

View full text Add to dashboard Cite

show abstract

“…Visualization of intermediate feature maps in our propsed ESCNet and the ESPNet [1] for input images in Cityscapes validation set. To implement visualization, we compressed feature maps into three dimensions using PCA, following [49].…”

Section: A Residual Connectionmentioning

confidence: 99%

“…7. To implement visualization, we compressed feature maps into three dimensions using PCA, following [49]. The first row in Fig.…”

Section: B Feature Map Visualizationmentioning

confidence: 99%

Efficient Semantic Segmentation Using Spatio-Channel Dilated Convolutions

Kim

Heo

2019

IEEE Access

View full text Add to dashboard Cite

There has been an increasing interest in reducing the computational cost to develop efficient deep convolutional neural networks (DCNN) for real-time semantic segmentation. In this paper, we introduce an efficient convolution method, Spatio-Channel dilated convolution (SCDC) which is composed of structured sparse kernels based on the principle of split-transform-merge. Specifically, it employs the kernels whose shapes are dilated, not only in spatial domain, but also in channel domain, using a channel sampling approach. Based on SCDC, we propose an efficient convolutional module named Efficient Spatio-Channel dilated convolution (ESC). With ESC modules, we further propose ESCNet based on ESPNet architecture which is one of the state-of-the-art real-time semantic segmentation network that can be easily deployed on edge devices. We evaluated our ESCNet on the Cityscapes dataset and obtained competitive results, with a good trade-off between accuracy and computational cost. The proposed ESCNet achieves 61.5 % mean intersection over union (IoU) with only 196 K network parameters, and processes high resolution images at a rate of 164 frames per second (FPS) on a standard Titan Xp GPU. Various experimental results show that our method is reasonably accurate, light, and fast. INDEX TERMS Semantic segmentation, spatio-channel dilated convolution, efficient neural network, real-time processing.

show abstract

“…Visual attention mechanisms has been successfully applied in various computer vision tasks such as image captioning [12], semantic segmentation [38], object detection [39] and facial trait classification [40]. For example, Tian et al [40] proposed a Fisher LDA based structured pruning approach to discard less informative filters of the final convolutional layer, and the approach achieves good accuracy with high efficiency.…”

Section: Related Workmentioning

confidence: 99%

Adaptive Weighted CNN Features Integration for Correlation Filter Tracking

Yang

2019

IEEE Access

View full text Add to dashboard Cite

Visual object tracking is an active and challenging research topic in computer vision, as objects often undergo significant appearance changes caused by occlusion, deformation, and background clutter. Although convolutional neural network (CNN)-based trackers have achieved competitive results, there are still some limitations. Most existing CNN-based trackers track the object by leveraging high-level semantic features of the highest convolutional layer, which may lead to low-spatial resolution feature maps and degrade the localization precision of tracking. Furthermore, these trackers hardly benefit from end-toend training since the extraction of features and the learning of classifier are separated. To deal with the above-mentioned issues, we design an adaptive weighted CNN features-based Siamese network for tracking. To capture spatial and semantic information of the object, we design a feature extraction network that derives feature maps by concatenating features of all convolutional layers. To make the features representation more discriminative, we propose a feature integration network. In the feature integration network, we propose a holistic-part network to capture strong visual cues and learn the semantic relations between the holistic object and its parts and combine the holistic-part network with spatial and channel attention mechanisms to adaptively assign weights to each region and channel of the feature maps. In addition, the designed Siamese network can be trained offline end-to-end. The experimental results on the benchmark datasets OTB50 and OTB100 demonstrate that the proposed tracker achieves favorable performance against several state-of-theart trackers while running at an average speed of 20.5 frames/s. INDEX TERMS Visual object tracking, correlation filter, Siamese convolutional neural network, feature integration, channel attention and spatial attention.

show abstract

Exploring New Backbone and Attention Module for Semantic Segmentation in Street Scenes

Cited by 21 publications

References 33 publications

Deep multimodal fusion for semantic image segmentation: A survey

Deep multimodal fusion for semantic image segmentation: A survey

Efficient Semantic Segmentation Using Spatio-Channel Dilated Convolutions

Adaptive Weighted CNN Features Integration for Correlation Filter Tracking

Contact Info

Product

Resources

About