Semantic object segmentation via detection in weakly labeled video

Abstract

Semantic object segmentation in video is an important step for large-scale multimedia analysis. In many cases, however, semantic objects are only tagged at video-level, making them difficult to be located and segmented. To address this problem, this paper proposes an approach to segment semantic objects in weakly labeled video via object detection. In our approach, a novel video segmentationby-detection framework is proposed, which first incorporates object and region detectors pre-trained on still images to generate a set of detection and segmentation proposals. Based on the noisy proposals, several object tracks are then initialized by solving a joint binary optimization problem with min-cost flow. As such tracks actually provide rough configurations of semantic objects, we thus refine the object segmentation while preserving the spatiotemporal consistency by inferring the shape likelihoods of pixels from the statistical information of tracks. Experimental results on Youtube-Objects dataset and SegTrack v2 dataset demonstrate that our method outperforms state-of-the-arts and shows impressive results.

Bringing Background into the Foreground: Making All Classes Equal in Weakly-Supervised Video Semantic Segmentation

“…In the same setting of multiple foreground vs. single background, several methods have proposed to rely on additional supervision. For instance, [63] relied on the CPMC [8] region detector, which has been trained from pixel-level annotations, to segment foreground from background. In [58] and [15], object proposal methods trained from pixel-level and bounding box annotations, respectively, were employed.…”

Video semantic object segmentation by self-adaptation of DCNN

“…We validate our proposed method on the Youtube-Object-Dataset (Prest et al, 2012;Jain and Grauman, 2014;Zhang et al, 2015). In experiments our model greatly improved the pre-trained model by mitigating its drawback even when we do not use the weak labels.…”

Unseen Object Segmentation in Videos via Transferable Representations

“…However, these methods rely on the quality of generated segment proposals and may produce inaccurate results when taking low-quality segments as the input. Zhang et al [44] propose to utilize object detectors integrated with object proposals to refine segmentations in videos. Furthermore, Tsai et al [40] develop a co-segmentation framework by linking object tracklets from all the videos and improve the result.…”

“…Compared to the baseline FCN model [20] used in our algorithm, there is a performance gain of 9%. In addition, while existing methods rely on training the segment classifier [34], integrating object proposals with detectors [44], co-segmentation via modeling relationships between videos [40], or self-paced fine-tuning [42], the proposed method utilizes a self-learning scheme to achieve better segmentation results. With the ResNet-101 architecture, we compare our method with DeepLab [2] and FSEG [12].…”