Joint Monocular 3D Vehicle Detection and Tracking

Abstract

Vehicle 3D extents and trajectories are critical cues for predicting the future location of vehicles and planning future agent ego-motion based on those predictions. In this paper, we propose a novel online framework for 3D vehicle detection and tracking from monocular videos. The framework can not only associate detections of vehicles in motion over time, but also estimate their complete 3D bounding box information from a sequence of 2D images captured on a moving platform. Our method leverages 3D box depth-ordering matching for robust instance association and utilizes 3D trajectory prediction for re-identification of occluded vehicles. We also design a motion learning module based on an LSTM for more accurate long-term motion extrapolation. Our experiments on simulation, KITTI, and Argoverse datasets show that our 3D tracking pipeline offers robust data association and tracking. On Argoverse, our imagebased method is significantly better for tracking 3D vehicles within 30 meters than the LiDAR-centric baseline methods.

MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation

“…Following methods continued to leverage Convolutional Neural Networks and focused only on Car instances. To regress 3D pose parameters from 2D detections, Deep3DBox [38], MonoGRnet [46], and Hu et al [23] used geometrical reasoning for 3D localization, while Multi-fusion [57] and ROI-10D [35] incorporated a module for depth estimation. Recently, Roddick et al [48] escaped the image domain by mapping image-based features into a birds-eye view representation using integral images.…”

A Self-Calibrating Probabilistic Framework for 3D Environment Perception Using Monocular Vision

“…Other solutions that provide detection and tracking rely on neural networks for the detection part, as well as for the tracking. In [21], the authors provide a method for tracking using Long Short Term Memory (LSTM) neural networks, but the main disadvantage is that it heavily depends on datasets and the availability of training data. The authors mention that the tracking is trained using imagery from realistic video games (synthetic data).…”

“…The solution from [21] employs a CNN for candidate region extraction, then uses another CNN for orientation and size estimation and, in the end, makes use of the LSTM neural network to track the detections. The main drawback is that it heavily relies on training data, the authors even mention that they used extensive synthetic images during the training and development of the approach.…”

A Viewpoint Aware Multi-Task Learning Framework for Fine-Grained Vehicle Recognition

“…I N the last decade, the widespread use of visual traffic surveillance systems has led to the rapid growth of video data that need to be processed. With the increasing amount of available video data, computer vision technology has been widely used in the field of intelligent transportation [1]- [3]. Vehicle fine-grained recognition has attracted more and more research interest [4]- [9], and its main goal is to identify the detailed information of a vehicle including make, model, submodel, etc.…”