Local Memory Read-and-Comparator for Video Object Segmentation

Abstract

Recently, the memory-based approach, which performs non-local matching between previously segmented frames and a query frame, has led to significant improvement in video object segmentation. However, the positional proximity of the target objects between the query and the local memory (previous frame), i.e. temporal smoothness, is often neglected. There are some attempts to solve the problem, but they are vulnerable and sensitive to large movements of target objects. In this paper, we propose local memory read-and-compare operations to address the problem. First, we propose local memory read and sequential local memory read modules to explore temporal smoothness between neighboring frames. Second, we propose the memory comparator to read the global memory and local memory adaptively by comparing the affinities of the global and local memories. Experimental results demonstrate that the proposed algorithm yields more strict segmentation results than the recent state-of-the-art algorithms. For example, the proposed algorithm improves the video object segmentation performance by 0.4% and 0.5% in terms of J &F on the most commonly used datasets, DAVIS2016 and DAVIS2017, respectively.

Video Mask Transfiner for High-Quality Video Instance Segmentation

“…We compare MaskFreeVIS with the state-of-the-art fully/weakly supervised methods on benchmarks YTVIS 2019/2021, OVIS and BDD100K MOTS. We integrate MaskFreeVIS on four representative methods [6,13,58,61], attaining consistent large gains over the strong baselines. YTVIS 2019/2021 Table 10 compares the performance on YTVIS 2019.…”

“…OVIS We also conduct experiments on OVIS in Table 12 using R50 as backbone. We integrate MaskFreeVIS with VITA [13], promoting the baseline performance from 12.1 to 15.7 under the mask-free training setting. BDD100K MOTS Table 13 further validates our approach on BDD100K MOTS.…”

“…One-stage works [4,27,32,63] adopt anchor-free detectors [50], generally using linear masks basis combination [3] or conditional mask prediction generation [49]. For the transformer-based models [6,13,47,58,64], VisTr [55] firstly adapts the transformer [5] for VIS, and IFC [16] further improves its efficiency via memory tokens. Seqformer [58] proposes frame query decompo-sition while Mask2Former [6] includes masked attention.…”

“…For the loss balance weights in Equation 7and Equation 8 of the paper, we set λ pair to 1.0 and λ temp to 0.1. We follow the same training setting and schedule of the baseline methods when integrating our TK-Loss with Mask2Former [6], SeqFormer [58], VITA [13] and Unicorn…”

Box Supervised Video Segmentation Proposal Network

“…Existing methodologies for VIS can be broadly categorized into offline and online methods. Offline methods [21,3,17,7,28] process the entire video at once. In contrast, online methods [27,12,20,4,11,22,8,6] process video sequences frame by frame.…”