Delving Deep into Many-to-many Attention for Few-shot Video Object Segmentation

Chen, Haoxin; Wu, Huawei; Zhao, Nanxuan; Ren, Sucheng; He, Shengfeng

doi:10.1109/cvpr46437.2021.01382

Cited by 12 publications

(33 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to video segmentation and few-shot segmentation with static imagery, there has been limited work on FS-VOS. Recent efforts focused on exploring attention [9], [10]. Co-attention conditioned on visual as well as semantic features was proposed and evaluated using a protocol that did not maintain the same support set on the entire sequence [9].…”

Section: Related Workmentioning

confidence: 99%

“…Similar to few-shot object segmentation, few-shot video object segmentation (FS-VOS) segments objects in query videos with novel classes specified by a support set of images. Compared to few-shot segmentation, FS-VOS has received limited attention [9], [10].…”

Section: Introductionmentioning

confidence: 99%

“…We also highlight two shortcomings with the evaluation protocol used in previous FS-VOS work [10]. First, the benchmark [10] does not provide exhaustively labelled masks, i.e. not all occurrences of a certain class are labelled.…”

Section: Introductionmentioning

confidence: 99%

“…This approach allows the well segmented frames in a video to guide the training of other frames. Moreover, it avoids expensive on-line backbone fine-tuning [10], yet yields superior results. (iii) We improve the previously proposed FS-VOS evaluation protocol by building an exhaustively labelled FS-VOS benchmark that operates crossdomain with data distribution shift, called PASCAL-to-MiniVSPW.…”

Section: Introductionmentioning

confidence: 99%

“…(iv) We propose a new score, k-shot stability, to assess the difficulty of the few-shot inference task for any approach and show that our spatiotemporal regularizers improve on the baseline in challenging scenarios. Moreover, we evaluate the temporal consistency of our predictions using a video consistency metric [20], which was lacking in previous work [10]. Our code and benchmarks are publicly available at https://github.com/MSiam/tti fsvos.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Temporal Transductive Inference for Few-Shot Video Object Segmentation

Siam¹,

Derpanis²,

Wildes³

2022

Preprint

View full text Add to dashboard Cite

Few-shot video object segmentation (FS-VOS) aims at segmenting video frames using a few labelled examples of classes not seen during initial training. In this paper, we present a simple but effective temporal transductive inference (TTI) approach that leverages temporal consistency in the unlabelled video frames during few-shot inference. Key to our approach is the use of both global and local temporal constraints. The objective of the global constraint is to learn consistent linear classifiers for novel classes across the image sequence, whereas the local constraint enforces the proportion of foreground/background regions in each frame to be coherent across a local temporal window. These constraints act as spatiotemporal regularizers during the transductive inference to increase temporal coherence and reduce overfitting on the few-shot support set. Empirically, our model outperforms state-of-the-art meta-learning approaches in terms of mean intersection over union on YouTube-VIS by 2.8%. In addition, we introduce improved benchmarks that are exhaustively labelled (i.e. all object occurrences are labelled, unlike the currently available), and present a more realistic evaluation paradigm that targets data distribution shift between training and testing sets. Our empirical results and in-depth analysis confirm the added benefits of the proposed spatiotemporal regularizers to improve temporal coherence and overcome certain overfitting scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%