STC: A Simple to Complex Framework for Weakly-Supervised Semantic Segmentation

Abstract

Abstract-Recently, significant improvement has been made on semantic object segmentation due to the development of deep convolutional neural networks (DCNNs). Training such a DCNN usually relies on a large number of images with pixel-level segmentation masks, and annotating these images is very costly in terms of both finance and human effort. In this paper, we propose a simple to complex (STC) framework in which only image-level annotations are utilized to learn DCNNs for semantic segmentation. Specifically, we first train an initial segmentation network called Initial-DCNN with the saliency maps of simple images (i.e., those with a single category of major object(s) and clean background). These saliency maps can be automatically obtained by existing bottom-up salient object detection techniques, where no supervision information is needed. Then, a better network called Enhanced-DCNN is learned with supervision from the predicted segmentation masks of simple images based on the Initial-DCNN as well as the image-level annotations. Finally, more pixel-level segmentation masks of complex images (two or more categories of objects with cluttered background), which are inferred by using Enhanced-DCNN and image-level annotations, are utilized as the supervision information to learn the Powerful-DCNN for semantic segmentation. Our method utilizes 40K simple images from Flickr.com and 10K complex images from PASCAL VOC for step-wisely boosting the segmentation network. Extensive experimental results on PASCAL VOC 2012 segmentation benchmark well demonstrate the superiority of the proposed STC framework compared with other state-of-the-arts.

Richer Convolutional Features for Edge Detection

“…It is usually considered as a low-level technique, and varieties of high-level tasks [8,27] have greatly benefited from the development of edge detection, such as object detection [18,57], object proposal [10,56,[62][63][64] and image segmentation [1,3,9,58].…”

Exploiting Saliency for Object Segmentation from Image Level Labels

“…For semantic labelling, different forms of supervision have been explored: image labels [32,31,30,33,46,18], points [3], scribbles [47,24], and bounding boxes [9,30,16]. In this work we focus on image labels as the main form of supervision.…”

“…Recognizing the ill-posed nature of the problem, [17] and [37] propose to collect user-feedback as additional information to guide the training of a segmentation convnet. The closest work to our approach is [46], which also uses saliency as a cue to improve weakly supervised semantic segmentation. There are however a number of differences.…”

“…Strongly co-occurring categories (such as train and rails, sculls and oars, snowbikes and snow) cannot be separated without additional information. Because additional information is needed to solve the task, previous work have explored different avenues, including class-specific size priors [31], crawling additional images [33,46], or requesting corrections from a human judge [17,37].…”

Scale-Aware Pixelwise Object Proposal Networks

“…CNN-based methods follow the great success of Convolutional Neural Network in other vision tasks, [23], [24], [25], [26], especially semantic segmentation [27], [28], [29]. They leverage the powerful discrimination ability of Convolutional Neural Network (CNN) to extract visual features as inputs of other techniques to produce proposals or directly regress the coordinates of all the object bounding boxes in an image.…”