MOTS: Multi-Object Tracking and Segmentation

Voigtlaender, Paul; Krause, Michael; Ošep, Aljoša; Luiten, Jonathon; Sekar, Berin Balachandar Gnana; Geiger, Andreas; Leibe, Bastian

doi:10.1109/cvpr.2019.00813

Cited by 557 publications

(537 citation statements)

References 59 publications

Supporting

Mentioning

536

Contrasting

Unclassified

Order By: Relevance

“…Accurate detection is still essential to cell tracking performance, and deep learning-based segmentation methods still make impactful errors as cells become more crowded. We expect this to be mitigated as more expansive sets of data are annotated, and as segmentation methods that use spatiotemporal information to inform segmentation decisions come online 42 . This method, and all supervised machine learning methods, is limited by the training data that empower it.…”

Section: Discussionmentioning

confidence: 99%

Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning

Moen

Borba

Miller

et al. 2019

Preprint

View full text Add to dashboard Cite

Live-cell imaging experiments have opened an exciting window into the behavior of living systems. While these experiments can produce rich data, the computational analysis of these datasets is challenging. Single-cell analysis requires that cells be accurately identified in each image and subsequently tracked over time. Increasingly, deep learning is being used to interpret microscopy image with single cell resolution. In this work, we apply deep learning to the problem of tracking single cells in live-cell imaging data. Using crowdsourcing and a human-in-the-loop approach to data annotation, we constructed a dataset of over 11,000 trajectories of cell nuclei that includes lineage information. Using this dataset, we successfully trained a deep learning model to perform cell tracking within a linear programming framework. Benchmarking tests demonstrate that our method achieves state-of-the-art performance on the task of cell tracking with respect to multiple accuracy metrics. Further, we show that our deep learning-based method generalizes to perform cell tracking for both fluorescent and brightfield images of the cell cytoplasm, despite having never been trained on those data types. This enables analysis of live-cell imaging data collected across imaging modalities. A persistent cloud deployment of our cell tracker is available at http://www.deepcell.org.

show abstract

Section: Discussionmentioning

confidence: 99%

Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning

Moen

Borba

Miller

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…We use the KITTI tracking benchmark to evaluate our tracking results for both cars and pedestrians in realworld driving scenes. The annotations for this have been extended with pixel-level mask annotations to evaluate the MOTS task (multi-object tracking and segmentation) [38]. We use the official KITTI test server as well as the validation split from [38] for evaluation.…”

Section: Methodsmentioning

confidence: 99%

“…Segmentation Tracking. Recently [38], [28], [20] segmentation masks have been exploited for improving tracking and creating pixel-accurate tracking results. These often use optical flow [16] or scene flow [41] to model the motion of each pixel.…”

Section: Related Workmentioning

confidence: 99%

Track to Reconstruct and Reconstruct to Track

Luiten

Fischer

Leibe

2020

IEEE Robot. Autom. Lett.

Self Cite

115

View full text Add to dashboard Cite

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstructionbased tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.

show abstract

“…In this context, several existing methods can be used, both LiDAR [11,42] or RGB based [44]. As we mainly target (automotive) real-time tracking scenarios, we constrain the pose transform prediction to translation on the ground plane (equivalent to the xy-plane in all experiments) and relative rotation around the z-axis.…”

Section: Methodsmentioning

confidence: 99%

“…The sequences are divided into training and validation sequences, as in [44]. Table 3: Registration results on Synth20, containing simulated scans of meshes of 20 different classes of ModelNet40 [46].…”

Section: Datasetsmentioning

confidence: 99%

AlignNet-3D: Fast Point Cloud Registration of Partially Observed Objects

GroB

Ošep

Leibe

2019

2019 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

Methods tackling multi-object tracking need to estimate the number of targets in the sensing area as well as to estimate their continuous state. While the majority of existing methods focus on data association, precise state (3D pose) estimation is often only coarsely estimated by approximating targets with centroids or (3D) bounding boxes. However, in automotive scenarios, motion perception of surrounding agents is critical and inaccuracies in the vehicle close-range can have catastrophic consequences. In this work, we focus on precise 3D track state estimation and propose a learning-based approach for object-centric relative motion estimation of partially observed objects. Instead of approximating targets with their centroids, our approach is capable of utilizing noisy 3D point segments of objects to estimate their motion. To that end, we propose a simple, yet effective and efficient network, AlignNet-3D, that learns to align point clouds. Our evaluation on two different datasets demonstrates that our method outperforms computationally expensive, global 3D registration methods while being significantly more efficient. We make our data, code, and models available at https://www.vision. rwth-aachen.de/page/alignnet.

show abstract

MOTS: Multi-Object Tracking and Segmentation

Cited by 557 publications

References 59 publications

Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning

Caliban: Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning

Track to Reconstruct and Reconstruct to Track

AlignNet-3D: Fast Point Cloud Registration of Partially Observed Objects

Contact Info

Product

Resources

About