MPPNet: Multi-frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection

Chen, Xuesong; Shi, Shaoshuai; Zhu, Benjin; Cheung, Ka Chun; Xu, Hang; Li, Hongsheng

doi:10.1007/978-3-031-20074-8_39

Cited by 50 publications

(37 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To resolve such limitations, 3D-MAN [31] first attempts to employ the attention mechanism to align different views of 3D objects and then exploits a memory bank to store and aggregate multi-frame features for long sequence. Recently, Offboard3D [18] and MPPNet [2] improve much the detection performance, where they associate the detected boxes from each frame of the sequence as proposal trajectories, and extract high-quality proposal features by sampling sequential point cloud on the trajectories.…”

Section: Related Workmentioning

confidence: 99%

“…Our MSF method also samples points from the sequence, but it differs from those methods with proposal trajectories [2,18] in that we only generate proposals on the current frame and propagate them to explore features in preceding frames. This makes our method much more efficient and favorable to online detection systems.…”

Section: Related Workmentioning

confidence: 99%

“…3D object detection [1,2,6,7,9,14,21,[27][28][29]36] is one of the key technologies in autonomous driving, which helps the vehicle to better understand the surrounding environment and make critical decisions in the downstream tasks. As an indispensable sensing device in autonomous driving systems, LiDAR collects 3D measurements of the scene in The "Detect-and-Fuse" framework extracts features from each frame of the sequence and then fuses them, while (b) our proposed "Motion-guided Sequential Fusion" (MSF) method generates proposals on the current frame and propagates them to preceding frames to explore useful contexts in the sequence.…”

Section: Introductionmentioning

confidence: 99%

“…Recent works mainly focus on deep feature fusion with multi-frame point clouds, for example, aggregating dense birds-eye-view features via Transformer models [31,37], passing the voxel features to LSTM [8] or GRU [32] modules for temporal modeling. Some top-performing detectors [2,18] focus on fusing proposal features, where a tracker is employed to associate the 3D proposals across frames, and a region-based network is applied to refine the current proposals by incorporating contextual features from the proposal trajectories. These approaches generally follow a "Detect-and-Fuse" framework, as shown in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…It is found that the existing point cloud pooling methods [2,19,23,30] are inefficient, taking more than 40 milliseconds when processing millions of points from sequential point clouds. We find that the major bottleneck lies in the heavy computation of pair-wise distances between n points and m proposals, which costs O(nm) complexity.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences

He¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Point cloud sequences are commonly used to accurately detect 3D objects in applications such as autonomous driving. Current top-performing multi-frame detectors mostly follow a Detect-and-Fuse framework, which extracts features from each frame of the sequence and fuses them to detect the objects in the current frame. However, this inevitably leads to redundant computation since adjacent frames are highly correlated. In this paper, we propose an efficient Motion-guided Sequential Fusion (MSF) method, which exploits the continuity of object motion to mine useful sequential contexts for object detection in the current frame. We first generate 3D proposals on the current frame and propagate them to preceding frames based on the estimated velocities. The points-of-interest are then pooled from the sequence and encoded as proposal features. A novel Bidirectional Feature Aggregation (BiFA) module is further proposed to facilitate the interactions of proposal features across frames. Besides, we optimize the point cloud pooling by a voxel-based sampling technique so that millions of points can be processed in several milliseconds. The proposed MSF method achieves not only better efficiency than other multi-frame detectors but also leading accuracy, with 83.12% and 78.30% mAP on the LEVEL1 and LEVEL2 test sets of Waymo Open Dataset, respectively. Codes can be found at https://github.com/skyhehe123/MSF.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences

He¹,

Li²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

PTSEFormer: Progressive Temporal-Spatial Enhanced TransFormer Towards Video Object Detection

Wang

Tang

Liu

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The Detection Transformer (DETR) has revolutionized the design of CNN-based object detection systems, showcasing impressive performance. However, its potential in the domain of multi-frame 3D object detection remains largely unexplored. In this paper, we present STEMD, a novel end-to-end framework for multi-frame 3D object detection based on the DETR-like paradigm. STEMD treats multi-frame 3D object detection as a sequence-to-sequence task and effectively captures spatial-temporal dependencies at both the feature and query levels. Specifically, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network, which represents queries as nodes in a graph and enables effective modeling of object interactions within a social context. To solve the problem of missing hard cases in the proposed output of the encoder in the current frame, we incorporate the output of the previous frame to initialize the query input of the decoder. Moreover, to mitigate the issue of redundant detection results, where the model generates numerous overlapping boxes from similar queries, we consider an IoU regularization term in the loss function, which can distinguish between queries matched with the ground-truth box and queries that are similar but unmatched during the refinement process, leading to reduced redundancy and more accurate detections. Through extensive experiments, we demonstrate the effectiveness of our approach in handling challenging scenarios, while incurring only a minor additional computational overhead. Our framework will potentially bring insights to this field. The code will be available at https://github.com/Eaphan/STEMD.

show abstract

RAPiD-Seg: Range-Aware Pointwise Distance Distribution Networks for 3D LiDAR Segmentation

Li,

Shum,

Breckon

2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

MPPNet: Multi-frame Feature Intertwining with Proxy Points for 3D Temporal Object Detection

Cited by 50 publications

References 35 publications

MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences

MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences

PTSEFormer: Progressive Temporal-Spatial Enhanced TransFormer Towards Video Object Detection

RAPiD-Seg: Range-Aware Pointwise Distance Distribution Networks for 3D LiDAR Segmentation

Contact Info

Product

Resources

About