Semantic segmentation via highly fused convolutional network with multiple soft cost functions

Abstract

Semantic image segmentation is one of the most challenged tasks in computer vision. In this paper, we propose a highly fused convolutional network, which consists of three parts: feature downsampling, combined feature upsampling and multiple predictions. We adopt a strategy of multiple steps of upsampling and combined feature maps in pooling layers with its corresponding unpooling layers. Then we bring out multiple pre-outputs, each pre-output is generated from an unpooling layer by one-step upsampling. Finally, we concatenate these pre-outputs to get the final output. As a result, our proposed network makes highly use of the feature information by fusing and reusing feature maps. In addition, when training our model, we add multiple soft cost functions on pre-outputs and final outputs. In this way, we can reduce the loss reduction when the loss is back propagated. We evaluate our model on three major segmentation datasets: CamVid, PASCAL VOC and ADE20K. We achieve a state-of-the-art performance on CamVid dataset, as well as considerable improvements on PASCAL VOC dataset and ADE20K dataset.Key Words: semantic segmentation, multiple soft cost functions, highly fused convolutional network series of CNN-based networks and some useful independent modules have been brought forward too, such as dropout [1] and batch normalization [2]. Convolutional networks are now leading many computer vision tasks, including image classification [3,4], object detection [5,6,7,8] and semantic image segmentation [9,10,11]. Image semantic segmentation is also known as scene parsing, which aims to classify every pixel in the image. It is one of the most challenged and primary tasks in computer vision. Network models for scene parsing task are always based on reliable models for image classification, since segmentation datasets have fewer images than the large available classification datasets. The landmark fully convolutional network (FCN) [9] for semantic segmentation is based on VGG-net [12], which is trained on the famous ImageNet dataset [13]. A novel end-to-end segmentation learning method is introduced in FCNs. In detail, convolution layers with a kernel size of 1x1 take the place of fully connected layers, followed by unpooling layers to recover the spatial resolution of the feature maps. As a consequence, output maps can achieve the same resolution as the input image of the models. In order to reduce the noise in output maps, FCN introduces skip connections between pooling layers and unpooling layers. Since the proposal of FCN, modern works on segmentation are mostly based on it [14,15].In our previous work, a fully combined convolutional network (FCCN) is explored to improve the segmentation performance [16]. We adopt a layer-by-layer upsampling method. After each upsampling operation, we acquire an output with the double size of the input feature maps. We also combine the corresponding pooling and unpooling layers. Another important work in FCCN is the soft cost function used for training the model. Evaluated on CamVid dataset...

UnetDVH-Linear: Linear Feature Segmentation by Dilated Convolution with Vertical and Horizontal Kernels

“…There are five upsampling blocks in the upsampling part that reserve feature information within different scales. We have the up-conv block proposed by Yang et al [ 36 ], named HF layers, which can fuse multi-scale upsampling information. The proposed VH-stage in [ 1 ] solves the problem that short kernels are not enough to cover full linear features, while long kernels may reduce the efficiency of the network, even reducing the segmentation performance.…”

“…Guided by dilated convolution [ 21 , 35 ] and the VH-stage, we designed the DVH block to improve the semantic segmentation network for linear feature extraction. HFCN [ 36 ] comes up with a further structured layer based on FCCN [ 37 ], and each unpooling layer follows a combination layer. This method can fuse upsampling features of different receptive fields in high-fusion layers.…”

VH-HFCN based Parking Slot and Lane Markings Segmentation on Panoramic Surround View

“…In prior work, based on FCN, we put forward efficient segmentation networks FCCN [22] and HFCN [23]. We proposed the cost function method to train our network.…”

“…5. Our network is based on HFCN [23], which improves the up-sampling process of the FCN model, divide the up-sampling operation on the feature maps into five steps. After each step, the feature maps expand to double size in both width and height, as a reverse operation of the previous pooling layer.…”

TiEV: The Tongji Intelligent Electric Vehicle in the Intelligent Vehicle Future Challenge of China

“…2) Surround vision: Based on the panoramic view ( Fig. 7 a)) fused by four fisheye cameras, TiEV introduces a VHstage to HFCN to robustly segment parking slots and lane markings under various scene and illumination conditions [21]. The results are shown in Fig.…”