End-to-End Referring Video Object Segmentation with Multimodal Transformers

Abstract

The referring video object segmentation task (RVOS) involves segmentation of a text-referred object instance in the frames of a given video. Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it. In this paper, we propose a simple Transformer-based approach to RVOS. Our framework, termed Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence prediction problem. Following recent advancements in computer vision and natural language processing, MTTR is based on the realization that video and text can both be processed together effectively and elegantly by a single multimodal Transformer model. MTTR is end-to-end trainable, free of text-related inductive bias components and requires no additional maskrefinement post-processing steps. As such, it simplifies the RVOS pipeline considerably compared to existing methods. Evaluation on standard benchmarks reveals that MTTR significantly outperforms previous art across multiple metrics. In particular, MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and JHMDB-Sentences datasets respectively, while processing 76 frames per second. In addition, we report strong results on the public validation set of Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the attention of researchers. The code to reproduce our experiments is available at https://github.com/mttr2021/MTTR.

R^2VOS: Robust Referring Video Object Segmentation via Relational Multimodal Cycle Consistency

“…ClawCraneNet [16] leverages cross-modal attention to bridge the semantic correlation between textual and visual modalities. ReferFormer [41] and MTTR [1] are two latest works that utilize transformers to decode or fuse multimodal features. ReferFormer [41] employs a linguistic prior to the transformer decoder to focus on the referred object.…”

“…ReferFormer [41] employs a linguistic prior to the transformer decoder to focus on the referred object. MTTR [1] leverages a multimodal transformer encoder to fuse linguistic and visual features. Different from other vision-language tasks, e.g., image-text retrieval [20,22,29] and video question answering [12,37], R-VOS needs to construct object-level multimodal semantic consensus in a dense visual representation.…”

“…Visual encoder. Following previous methods [1,41,40], we build the visual encoder with a visual backbone and a deformable transformer encoder [46] on top of it. The extracted features from the backbone are flattened, projected to a lower dimension, added with positional encoding [9], and then fed into a deformable transformer encoder [46] similar to the previous method [41].…”

“…Previous works [1,41] tackle the R-VOS problem with a strong assumption that the referred object exists in the video, i.e., there is an object-level semantic consensus between the expression and the video. However, this assumption does not always hold in practice.…”

“…Even when semantic consensus exists in the given video-language pairs, it is still challenging to locate the correct object in the video due to the multimodal nature of the R-VOS task. Recently, MTTR [1] employs a multimodal transformer encoder to learn a joint representation of the linguistic expression and video, and then obtains the referred object by ranking all objects in the video. ReferFormer [41] follows the image-level method, ReTR [14], to adopt the linguistic expression as a query to the transformer decoder to avoid redundant ranking of all objects.…”