YouTube-VOS: Sequence-to-Sequence Video Object Segmentation

Abstract

Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatialtemporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 4,453 YouTube video clips and 94 object categories. This is by far the largest video object segmentation dataset to our knowledge and has been released at http://youtube-vos.org. We further evaluate several existing state-of-the-art video object segmentation algorithms on this dataset which aims to establish baselines for the development of new algorithms in the future.

CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

“…Datasets We evaluate our method on two video object segmentation datasets: Youtube-VOS [37] and DAVIS-2017 [25]. Training The network is trained using the objective function described in 3.4.…”

“…Semi-supervised video object segmentation: Earlier works in video object segmentation used hand-crafted features based on appearance, boundary and optical flow [1,9,15,27,23]. The availability of large-scale video object segmentation datasets [25,37] enabled us to explore deep learning methods for this problem. Most of the early works are mainly motivated by the image segmentation methods [3,35,20].…”

“…This is a challenging problem because of issues like occlusion, changes in object appearance over time, motion blur, fast motions, and scale variations of different objects. Deep learning approaches have achieved impressive results and the recent release of the Youtube-VOS dataset [37] has allowed for the training and evaluation of new methods on a wider variety of videos and objects.…”

“…The majority of current approaches can be divided into two categories. The first are detection-based methods [2,4,14] that learn representations of the object segmented in the first frame and attempt to perform the pixel-wise detection of this object in future frames; the second is propagationbased methods [7,12,28,33,36] that formulate the task as a tracking problem and attempt to propagate the mask to fit the object over time. The first set of methods tends to segment single frames independently and rarely employ temporal information, while the later set segments single frames sequentially and makes use of temporal information, usually in the form of optical flow or RNNs.…”

DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation

“…Video object and instance segmentation problems have received significant attention [1,35,25] attributed to the recent availability of high-quality datasets, e.g., YouTube-VOS [45], DAVIS [29,30]. Given an input video, the aim is to separate the objects or instances from the background at the pixel-level.…”