Video Segmentation by Tracking Many Figure-Ground Segments

Abstract

We propose an unsupervised video segmentation approach by simultaneously tracking multiple holistic figureground segments. Segment tracks are initialized from a pool of segment proposals generated from a figure-ground segmentation algorithm. Then, online non-local appearance models are trained incrementally for each track using a multi-output regularized least squares formulation. By using the same set of training examples for all segment tracks, a computational trick allows us to track hundreds of segment tracks efficiently, as well as perform optimal online updates in closed-form. Besides, a new composite statistical inference approach is proposed for refining the obtained segment tracks, which breaks down the initial segment proposals and recombines for better ones by utilizing highorder statistic estimates from the appearance model and enforcing temporal consistency. For evaluating the algorithm, a dataset, SegTrack v2, is collected with about 1,000 frames with pixel-level annotations. The proposed framework outperforms state-of-the-art approaches in the dataset, showing its efficiency and robustness to challenges in different video sequences.

Supervoxel-Consistent Foreground Propagation in Video

“…Note that unsupervised multiplehypothesis methods [17,19,21,39] are not comparable in this semi-supervised single-hypothesis setting. We also test the following baselines:…”

“…Several recent algorithms aim to upgrade bottom-up video segmentation to object-level segments [17,19,21,22,39]. While the details vary, the main idea is to generate foreground object hypotheses per frame using learned models of "object-like" regions (e.g., salient, convex, distinct motion from background), and then optimize their temporal connections to generate space-time tubes.…”

“…The propagation paradigm is a compelling middle ground. First, it removes ambiguity about what object is of interest, which, despite impressive advances [17,19,21,39], remains an inherent pitfall for unsupervised methods. Accordingly, the propagation setting can accommodate a broader class of videos, e.g., those in which the object does not move much, or shares appearance with the background.…”

“…At one extreme, there are purely unsupervised methods that produce coherent space-time regions from the bottom up, without any video-specific labels [8,12,14,17,19,21,36,38,39]. At the other extreme, there are strongly supervised interactive methods, which require a human in the loop to correct the system's errors [4,10,20,25,34,35].…”

Gamifying Video Object Segmentation

“…• SegTrack v2 [51]: we created only one level by incorporating a subset of videos from the SegTrack v2 dataset (namely, birdfall, bmx, cheetah, drift, hummingbird, monkey and monkeydog) into a single sequence, since many of them were just few dozen frames long. We favored sequences with multiple objects, and excluded videos where the target, though moving, appeared at the same frame location due to camera motion (e.g.…”