Tackling Background Distraction in Video Object Segmentation

Abstract

Semi-supervised video object segmentation (VOS) aims to densely track certain designated objects in videos. One of the main challenges in this task is the existence of background distractors that appear similar to the target objects. We propose three novel strategies to suppress such distractors: 1) a spatio-temporally diversified template construction scheme to obtain generalized properties of the target objects; 2) a learnable distance-scoring function to exclude spatially-distant distractors by exploiting the temporal consistency between two consecutive frames; 3) swap-and-attach augmentation to force each object to have unique features by providing training samples containing entangled objects. On all public benchmark datasets, our model achieves a comparable performance to contemporary state-of-the-art approaches, even with real-time performance. Qualitative results also demonstrate the superiority of our approach over existing methods. We believe our approach will be widely used for future VOS research. Code and models are available at https://github.com/suhwan-cho/TBD.

One-Shot Video Inpainting

“…We quantitatively compare our method to existing twostage methods in Table 1. For the VOS models, we adopt FRTM (Robinson et al 2020), CFBI (Yang, Wei, and Yang 2020), BMVOS (Cho et al 2022a), and TBD (Cho et al 2022b). For the VI models, CPNet (Lee et al 2019), STTN (Zeng, Fu, and Chao 2020), FGVC (Gao et al 2020), and FuseFormer (Liu et al 2021) are used.…”

“…We adopt two datasets for network training, COCO (Lin et al 2014) and YouTube-VOS 2018 (Xu et al 2018). As COCO is an image dataset, we randomly augment each image to generate videos, following the protocol in STM (Oh et al 2019a) and TBD (Cho et al 2022b). We resize all training videos to a 240 × 432 resolution and use them as clean videos.…”

Pixel-Level Bijective Matching for Video Object Segmentation

“…Considering the lack of details when only employing high-level feature matching, HMMN [36] proposes a novel hierarchical matching mechanism to capture small objects as well. To relieve potential errors that can be caused by employing a pixel-level template, AOC [46] employs an adaptive proxylevel template, and TBD [6] employs both pixel-level and object-level templates simultaneously.…”

“…Decoder. The decoder is designed identically to TBD [6]. It consists of convolutional layers that fuse and refine different features, and deconvolutional layers [49] that upscale the refined features.…”

“…In all training stages, video frames are randomly cropped to have 384×384 resolution. For stable network training, balanced random cropping is adopted as in CFBI [47] and TBD [6]. To provide challenging but natural training samples, the swap-and-attach augmentation is applied with a probability of 20%, as in TBD.…”