Track, Then Decide: Category-Agnostic Vision-Based Multi-Object Tracking

Abstract

The most common paradigm for vision-based multi-object tracking is tracking-by-detection, due to the availability of reliable detectors for several important object categories such as cars and pedestrians. However, future mobile systems will need a capability to cope with rich human-made environments, in which obtaining detectors for every possible object category would be infeasible. In this paper, we propose a model-free multi-object tracking approach that uses a categoryagnostic image segmentation method to track objects. We present an efficient segmentation mask-based tracker which associates pixel-precise masks reported by the segmentation. Our approach can utilize semantic information whenever it is available for classifying objects at the track level, while retaining the capability to track generic unknown objects in the absence of such information. We demonstrate experimentally that our approach achieves performance comparable to stateof-the-art tracking-by-detection methods for popular object categories such as cars and pedestrians. Additionally, we show that the proposed method can discover and robustly track a large variety of other objects.

Track to Reconstruct and Reconstruct to Track

“…Segmentation Tracking: Comparison to state-of-theart. It also significantly outperforms other tracking methods [27] and [28] using the same detections and segmentations. In Table II we also evaluate the best versions of MOTSFusion on the MOTS test server, where we significantly outperform the previous best results from [38].…”

Large-Scale Object Mining for Object Discovery from Unlabeled Video

“…For tracking, we build upon our recent work and utilize our category-agnostic multi-object tracker (CAMOT) [32]. In a nutshell, using this tracker object candidates are obtained as follows (see Fig.…”

“…Tracks are thus automatically labeled by the recognized category type (i.e., as one of the COCO [25] categories) or as unknown object track. Finally, for each frame, a mutually consistent subset of tracks is picked by performing MAP inference using a conditional random field (CRF) model (for details, see [32]). This way, we obtain a reduced set of object tracks.…”

“…Since model selection is performed on a per-frame basis, one object may be split into several short tracklets. As a final postprocessing step extending our tracker [32] we progressively merge short selected tracklets into final tracks. In each frame, either a) existing tracklet h i is re-selected and trivially continues an existing track H k , or b) tracklet h i is not re-selected and its track is continued by another selected tracklet h j if they have a sufficient overlap.…”

“…For both we use stereo for estimating depth [11]. Compared to the original CAMOT [32], we replace the dense scene flow [49] by a sparse scene flow [24] for initialization. Sparse scene flow is less accurate, but has a lower processing time by several orders of magnitude.…”