BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation

Abstract

The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for real-time semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the pro-posed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048×1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy. Code and trained models will be made publicly available.

Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes

“…The encoder adopts two new operations (i.e., channel split and shuffle) in each residual block to significantly reduce the computational cost, while the decoder uses an attention pyramid network to further reduce the network complexity. Bilateral Segmentation Network (BiSeNet) [32] designs the spatial path and the semantic path to respectively preserve the spatial information and obtain the sufficient receptive field for improving the accuracy and speed of semantic segmentation. Based on the two paths, a new Feature Fusion Module (FFM) is developed to combine the features efficiently.…”

“…Therefore, LBN-AA is able to generate a dense sampling by concatenating the feature maps from these blocks, where the neighboring information from different blocks can complement each other [19]. It is worth noting that BiSeNet [32] also employs an attention module called Attention Refinement Module (ARM) for semantic segmentation. However, CAM and ARM are significantly different.…”

“…To illustrate this dilemma, Fig. 1 gives the accuracy (mIoU) and inference speed (frames per second (fps)) obtained by several state-of-the-art methods, including FCN-8s [9], CRF-RNN [17], DeepLab [10], DeepLabv2 [12], DeepLabv3+ [13], ResNet-38 [18], PSPNet [11], DUC [19], RefineNet [20], LRR [21], DPN [22], FRRN [23], TwoColumn [24], SegNet [25], SQNet [26], ENet [27], arXiv:2003.08736v2 [cs.CV] 3 Apr 2020 ERFNet [28], ICNet [29], SwiftNetRN [30], LEDNet [31], BiSeNet1 [32], BiSeNet2 [32], DFANet [33] and our proposed method, on the Cityscapes test dataset. Clearly, how to achieve a good tradeoff between accuracy and speed is still a challenging problem.…”

A segmentation network with multiattention and its application to SAR image analysis

“…The point of the proposed model is to apply global attention module (GAM) and attention refinement module (ARM). The GAMs can capture the relationship between pixels and the global feature maps, while the ARM can capture the channel information to refine the detail of input feature.…”

“…The point of the proposed model is to apply global attention module (GAM) and attention refinement module (ARM). [3] The GAMs can capture the relationship between pixels a Correspondence to: Jianwu Long. E-mail: jwlong@cqut.edu.cn *College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, China **Graduate School of Sciences and Technology for Innovation, Yamaguchi University, Tokiwadai 2-16-1, Ube, Yamaguchi 755-8611, Japan and the global feature maps, while the ARM can capture the channel information to refine the detail of input feature.…”