Boundary-Aware CNN for Semantic Segmentation

Abstract

Semantic segmentation has always been a fundamental and critical task to scene understanding. Current deep convolutional neural networks (DCNN) are able to successfully learn context from very large receptive fields due to convolutions with deep layers. However, current convolutions in DCNNs does not consider local object boundaries that are the borders among different semantic regions. Convolution with equal contribution on the pixels across the boundary may lead to inferior segmentation results. In this paper, a novel boundary-aware convolution is proposed. It is able to effectively fuse features by adaptively assigning contributions from pixels within receptive fields according to the boundary similarity map. A new semantic segmentation network based on classical FCN8S is then designed by employing multi-scale boundary-aware convolution. The whole network is implemented end-to-end and evaluated with heterogeneous RGB and depth input. Experiments conducted on multiple datasets show that our boundary-aware CNN can effectively improve the semantic segmentation performance.

Simultaneous Semantic Segmentation and Depth Completion with Constraint of Boundary

“…Besides the feature maps from the encoder, the semantic and depth decoder branches also share the multi-scale skipped features and the boundary feature maps, as shown in Figure 3a. The core modules for both branches are the multiple UpBlocks, where the boundary feature B i is absorbed to construct a special boundary-aware convolution layer BaConv [20], as illustrated in Figure 3b. With the guidance of the boundary, boundary-aware convolution could focus more on the regions with similar semantic features and gather the contributions more adaptively to produce the output.…”

“…The quantitative verification results for the semantic segmentation task are shown in Table 4, where IoU_cat and IoU_cla represent mIoU corresponding to all of the 7 categories and 19 classes, respectively; fwt represents the running time of each frame in seconds. With the help of the multi-task learning framework, the proposed SSDNet model performs better than the baseline BaCNN [20] and most of its counterparts. Compared with the state-of-the-art real-time CNN methods such as encoder-decoder based ENet model [41] and its loss-edited version [42], ESPNet [43] and two-branch-fusion Fast-SCNN [44], our full model performs better in IoU without costing more time.…”

“…BaCNN: Baseline model proposed [20]. BaCNN is based on the backbone of FCN8S [5], and modified by replacing the first layer of each Conv_Block into the boundary-aware convolution.…”

“…With adding of the boundary and depth task, the performance is gradually improved. BaCNN [20] introduces boundary detection task and replaces the standard convolution with the boundary-aware convolution. SSDNet_Sem achieves further improvement than the baseline BaCNN model.…”

“…This also proves that the proposed multi-task joint network can promote the performance of every single task. In an order of top-down: the first column shows the RGB color image, the depth ground truth and the semantic segmentation ground truth; the second column shows the boundary-similarity result, the depth completion output, the semantic segmentation result, and corresponding BaCNN [20] result respectively.…”