Opening up Open World Tracking

Liu, Yang; Zulfikar, Idil Esen; Luiten, Jonathon; Dave, Achal; Ramanan, Deva; Leibe, Bastian; Ošep, Aljoša; Leal-Taixé, Laura

doi:10.1109/cvpr52688.2022.01846

Cited by 30 publications

(12 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the maturation of tracking and VIS tasks, recently introduced datasets have started to emphasize more challenging scenarios like tracking of objects of unknown category and tracking through heavy occlusions. The TAO-OW dataset [31] builds an open-world MOT benchmark on top of the longtailed TAO dataset [15] by introducing a set of "unknown" class labels which are not included in the training set. Both TAO and TAO-OW are not exhaustively labeled.…”

Section: Related Workmentioning

confidence: 99%

Robust Online Video Instance Segmentation with Track Queries

Zhan¹,

McKee²,

Lazebnik³

2022

Preprint

View full text Add to dashboard Cite

Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like UVO and OVIS. We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (RO-VIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multiobject tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

Robust Online Video Instance Segmentation with Track Queries

Zhan¹,

McKee²,

Lazebnik³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other works have used class agnostic localizers [11,51] to perform MOT on arbitrary objects. Recently, Liu et al [39] defined open-world tracking, a task that focuses on the evaluation of previously unseen objects. In particular, it requires any-object tracking as a stage that precedes object classification.…”

Section: Deploymentmentioning

confidence: 99%

“…Existing works perform scene segmentation and class agnostic tracking before classification [46,48,50] or utilize class-agnostic proposal generation [11,51], similar to open-world detection methods. Liu et al [39] propose an open-world tracking benchmark, TAO-OW, that evaluates class-agnostic tracking as a task that precedes classification. However, this comes with the limitation that the evaluation only captures tracker recall and no classification accuracy.…”

Section: Related Workmentioning

confidence: 99%

“…With respect to evaluation, the advantage of defining C novel is that we can apply closed-set tracking metrics in a straightforward manner, while open-world MOT [39] needs to resort to recall-based evaluation. Further, previous works [36,39] have shown that the official evaluation metric in TAO, Track mAP [73], is sub-optimal in terms of handling FPs in presence of missing annotations. On the contrary, the recently proposed TETA metric [36] handles this shortcoming via local cluster evaluation.…”

Section: Benchmarkmentioning

confidence: 99%

See 1 more Smart Citation

Tracking Every Thing in the Wild

Liu

Danelljan

Ding

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The ability to recognize, localize and track dynamic objects in a scene is fundamental to many real-world applications, such as self-driving and robotic systems. Yet, traditional multiple object tracking (MOT) benchmarks rely only on a few object categories that hardly represent the multitude of possible objects that are encountered in the real world. This leaves contemporary MOT methods limited to a small set of pre-defined object categories. In this paper, we address this limitation by tackling a novel task, openvocabulary MOT, that aims to evaluate tracking beyond predefined training categories. We further develop OVTrack, an open-vocabulary tracker that is capable of tracking arbitrary object classes. Its design is based on two key ingredients: First, leveraging vision-language models for both classification and association via knowledge distillation; second, a data hallucination strategy for robust appearance feature learning from denoising diffusion probabilistic models. The result is an extremely data-efficient open-vocabulary tracker that sets a new state-of-the-art on the large-scale, largevocabulary TAO benchmark, while being trained solely on static images.

show abstract

“…These include statistic-based methods [71,65], visual similarity-based clustering methods [29,26,40], linkage analysis [42] with appearance and geometric consistency [15,84,85,86], visual saliency [105,39], and unsupervised feature learning using deep generative models [44,72,63,3]. In contrast, unsupervised object detection from LiDAR sequences is fairly under-explored [18,94,78,48]. [18,57] proposed to sequentially update the detections and perform tracking based on motion cues from 3D LiDAR scans.…”

Section: Related Workmentioning

confidence: 99%

Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving

Najibi¹,

Ji²,

Yin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning-based perception and prediction modules in modern autonomous driving systems typically rely on expensive human annotation and are designed to perceive only a handful of predefined object categories. This closed-set paradigm is insufficient for the safety-critical autonomous driving task, where the autonomous vehicle needs to process arbitrarily many types of traffic participants and their motion behaviors in a highly dynamic world. To address this difficulty, this paper pioneers a novel and challenging direction, i.e., training perception and prediction models to understand open-set moving objects, with no human supervision. Our proposed framework uses self-learned flow to trigger an automated meta labeling pipeline to achieve automatic supervision. 3D detection experiments on the Waymo Open Dataset show that our method significantly outperforms classical unsupervised approaches and is even competitive to the counterpart with supervised scene flow. We further show that our approach generates highly promising results in open-set 3D detection and trajectory prediction, confirming its potential in closing the safety gap of fully supervised systems.

show abstract

Opening up Open World Tracking

Cited by 30 publications

References 61 publications

Robust Online Video Instance Segmentation with Track Queries

Robust Online Video Instance Segmentation with Track Queries

Tracking Every Thing in the Wild

Motion Inspired Unsupervised Perception and Prediction in Autonomous Driving

Contact Info

Product

Resources

About