Object Discovery in Videos as Foreground Motion Clustering

Abstract

We consider the problem of providing dense segmentation masks for object discovery in videos. We formulate the object discovery problem as foreground motion clustering, where the goal is to cluster foreground pixels in videos into different objects. We introduce a novel pixel-trajectory recurrent neural network that learns feature embeddings of foreground pixel trajectories linked across time. By clustering the pixel trajectories using the learned feature embeddings, our method establishes correspondences between foreground object masks across video frames. To demonstrate the effectiveness of our framework for object discovery, we conduct experiments on commonly used datasets for motion segmentation, where we achieve state-of-the-art performance. arXiv:1812.02772v2 [cs.CV] 5 Apr 2019 Encoder-Decoder: Y-NetLet I t ∈ R H×W ×3 , F t ∈ R H×W ×2 be an RGB image and forward optical flow image at time t, respectively. Our

Towards Segmenting Anything That Moves

“…For a fair comparison, we disable this component, applying tracking directly to the output of our two-stream model that detects only moving objects in each frame. Table 11 shows that our method significantly reduces the number of false positive segmentations compared to both [45,5] as evidenced by the improvement in ∆ Obj of nearly 65% (from 4 to 1.4), while performing competitively with [45] on F-measure.…”

U2-ONet: A Two-Level Nested Octave U-Structure Network with a Multi-Scale Attention Mechanism for Moving Object Segmentation

“…As shown in Table 5, the proposed model performs the best in recall. In terms of recall and F-measure, it outperforms CCG [18], OBV [19], and STB [50] by over 16.5% and 6.4%, respectively. The qualitative results are shown in Figure 6.…”

“…However, these methods cannot segment each moving object instance. More recent approaches have used optical-flow-based methods for the instance-level moving object segmentation, including a hierarchical motion segmentation system that combines geometric knowledge with a modern CNN for appearance modeling [18], a novel pixel-trajectory recurrent neural network to cluster foreground pixels in videos into different objects [19], a two-stream architecture to separately process motion and appearance [20], a new submodular optimization process to achieve trajectory clustering [50], and a statistical-inference-based method for the combination of motion and semantic cues [21]. In comparison, we propose a two-level nested U-structure deep network with octave convolution to segment each moving object instance while reducing the spatial redundancy and memory cost in the CNN.…”

“…Since some of the methods compared only provide metrics for the multi-object motion segmentation task, only the metrics for this task are compared. In order to compare with the other methods, to indicate the accuracy of the segmented object count, the ∆Obj metric was added, as defined in [19]. As shown in Table 5, the proposed model performs the best in recall.…”

“…Meanwhile, these methods are not able to segment new objects that have not been labeled in the training data. More recent approaches combine instance/semantic segmentation results with motion information from optical flow to segment moving object instances in dynamic scenes, as in [18][19][20][21]. These methods [19,20] can segment never-before-seen objects that have not been predefined in the training set based on their motion.…”