Opening up Open-World Tracking

Abstract

In this paper, we propose and study Open-World Tracking (OWT). Open-world tracking goes beyond current multiobject tracking benchmarks and methods which focus on tracking object classes that belong to a predefined closedset of frequently observed object classes. In OWT, we relax this assumption: we may encounter objects at inference time that were not labeled for training. The main contribution of this paper is the formalization of the OWT task, along with an evaluation protocol and metric (Open-World Tracking Accuracy, OWTA), which decomposes into two intuitive terms, one for measuring recall, and another for measuring track association accuracy. This allows us to perform a rigorous evaluation of several different baselines that follow design patterns proposed in the multi-object tracking community. Further we show that our Open-World Tracking Baseline, while performing well in the OWT setting, also achieves near state-of-the-art results on traditional closedworld benchmarks, without any adjustments or tuning. We believe that this paper is an initial step towards studying multi-object tracking in the open world, a task of crucial importance for future intelligent agents that will need to understand, react to, and learn from, an infinite variety of objects that can appear in an open world.

Keep the Human in the Loop: Arguments for Human Assistance in the Synthesis of Simulation Data for Robot Training

“…Noisy segmentations result in coarse boundaries between entities in the scenes, and in some cases, identify ghost objects that are not present in the scene. This is also due to algorithms relying on databases, which remain incomplete [43]. Once all object points have been identified, object meshes can be generated using mesh triangulation algorithms to enable interaction with virtual objects.…”

Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

“…While the long-tailed open world framework has been recieving increasing attention within detection (e.g., Liu et al, 2019;Joseph et al, 2021;Saito et al, 2022;Konan et al, 2022), segmentation (e.g., , and tracking (e.g., Liu et al, 2022;Dave, 2021), open-world prediction is still unexplored territory. One method that may be directly applicable to action prediction tasks is that proposed in Liu et al (2019), which uses contrastive learning and a memory framework to improve performance on both common and uncommon classes, and recognize examples that don't fall into any of the given classes.…”

Tracking Every Thing in the Wild

“…Recently, the most common paradigm for MOT is tracking-by-detection, focusing on learning better appearance features to strengthen association [22,33,49,29,31,26], modeling the displacement of each tracked object [2,51,40], or using a graph-based approach [44,5]. Previous MOT approaches mostly focus on benchmarks with a few common categories, while recent works [27,10] study the MOT in open-set settings where the goal is to track and segment any objects regardless of their categories. Those methods use a class agnostic trained detector or RPN network to generate object proposals, while classification is essential in many applications, e.g., video analysis.…”

“…Recently, Higher-Order Tracking Accuracy (HOTA) [30] was proposed to fairly balance both components by computing a separate score for each. Liu et al [27] proposes a recall-based evaluation to extend MOT into open-world settings. All above metrics do not independently access the classification performance, making them unsuitable for large-scale multi-category MOT.…”