Robust Online Video Instance Segmentation with Track Queries

Abstract

Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like UVO and OVIS. We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (RO-VIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multiobject tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos 1 .

Box Supervised Video Segmentation Proposal Network

“…However, tracking in complex videos presents significant challenges, such as short or long-term occlusion, instance appearance or disappearance, and ID switching. We observe that the propagation strategy of [20,11,29] is helpful when consecutive frames share a considerable similarity. However, in abrupt changes, propagation might accumulate errors like lost tracks, ID switches, or inconsistent masks.…”

“…The recent emergence of datasets [16,24] containing lengthy and occluded videos has presented more challenging, real-world scenarios for VIS, driving the development of advanced deployable online models, particularly Detection-Transformer (DETR) [2,30] variants. These models primarily rely on frame-level detection and inter-frame association facilitated through tracking [22,8] or query propagation [20,11,6,29].…”

“…Recent methods [8,22] exploited this characteristic to enhance instance similarity or association between frames. In contrast, several methods [11,6,29] continually represent evolving objects in videos by propagating instance queries from frame to frame.…”

“…TrackFormer [15] initially demonstrated this powerful technique on Multi-Object Tracking and Segmentation [18] challenge. Later on, this method was adopted by InstanceFormer [11], and ROVIS [29] for the VIS datasets. InstanceFormer propagated instance queries and reference points of Deformable-DETR [30] from frame to frame.…”