Joint 3D Object Detection and Tracking Using Spatio-Temporal Representation of Camera Image and LiDAR Point Clouds

Abstract

In this paper, we propose a new joint object detection and tracking (JoDT) framework for 3D object detection and tracking based on camera and LiDAR sensors. The proposed method, referred to as 3D DetecTrack, enables the detector and tracker to cooperate to generate a spatio-temporal representation of the camera and LiDAR data, with which 3D object detection and tracking are then performed. The detector constructs the spatio-temporal features via the weighted temporal aggregation of the spatial features obtained by the camera and LiDAR fusion. Then, the detector reconfigures the initial detection results using information from the tracklets maintained up to the previous time step. Based on the spatio-temporal features generated by the detector, the tracker associates the detected objects with previously tracked objects using a graph neural network (GNN). We devise a fully-connected GNN facilitated by a combination of rule-based edge pruning and attention-based edge gating, which exploits both spatial and temporal object contexts to improve tracking performance. The experiments conducted on both KITTI and nuScenes benchmarks demonstrate that the proposed 3D DetecTrack achieves significant improvements in both detection and tracking performances over baseline methods and achieves state-of-the-art performance among existing methods through collaboration between the detector and tracker.

msLPCC: A Multimodal-Driven Scalable Framework for Deep LiDAR Point Cloud Compression

“…Intuitively, taking the location information as an example, depth features represent the relative location information while point features represent the absolute location information, and it is important to weigh their contributions to the information fusion accordingly. To achieve this, we adopt an attentive block in (You et al 2019;Koh et al 2022), which adaptively fuses the location information between the depth and point features. The fused feature f k d(l) of the location information is obtained through cross-attentive addition, which can be formulated as:…”

Multi-Scene Generalized Trajectory Global Graph Solver with Composite Nodes for Multiple Object Tracking

“…They use object detection as object nodes and represent edges as possible trajectory hypotheses. In contrast to trackers that use neighbor frame information, graph-based approaches define the cross-frame object association problem as a global combinatorial optimization problem (Koh et al 2022;Dendorfer et al 2020a;Brasó and Leal-Taixé 2020;Zeng et al 2022). To this end, many studies have used different optimization strategies, including multi-cuts (Tang et al 2017), minimal cliques (Zamir, Dehghan, and Shah 2012), network flow (Berclaz et al 2011;Butt and Collins 2013), and disjoint path approaches (Hornáková et al 2020(Hornáková et al , 2021.…”

LiDAR-based 3D object detection and tracking for autonomous driving

“…MPPNet [74] employs RoI grid pooling to extract features from proposals and create trajectories for temporal fusion. MGTANet [75] utilizes an SM-VFE module for encoding features and leverages a motion-guided deformable module to align and merge multi-frame features. D-align [76] designs a dual-query attention network to leverage both target frame features and support frame features.…”