VPFNet: Improving 3D Object Detection With Virtual Point Based LiDAR and Stereo Data Fusion

Zhu, Hanqi; Deng, Jiajun; Ji, Jianmin; Mao, Qiuyu; Li, Houqiang; Zhang, Yanyong

doi:10.1109/tmm.2022.3189778

Cited by 85 publications

(15 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In order to construct a more robust system, there is an urgent need to collect collaborative perception data in complex environments and propose well-designed methods for various complex scenarios. Multi-sensor fusion helps compensate for weather and distance's effects on data quality, and virtual point cloud generation [108,109,110] can be used to predict long-range objects. Additionally, spatio-temporal data fusion is required to predict the trajectory of objects moving at high speeds, and these are promising directions for future research.…”

Section: B Collaboration Perception In Complex Scenesmentioning

confidence: 99%

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

Han¹,

Zhang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Section: B Collaboration Perception In Complex Scenesmentioning

confidence: 99%

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

Han¹,

Zhang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

“…Another solution based on LIDAR and multiview representations is proposed by [ 4 ]. They published a new architecture called VPFNet that aligns and aggregates the image data and point cloud at virtual points that can bridge the resolution gap between the two sensors (LIDAR and multiview images).…”

Section: Related Workmentioning

confidence: 99%

3D Vehicle Detection and Segmentation Based on EfficientNetB3 and CenterNet Residual Blocks

Kashevnik

Ali

2022

Sensors

View full text Add to dashboard Cite

In this paper, we present a two stages solution to 3D vehicle detection and segmentation. The first stage depends on the combination of EfficientNetB3 architecture with multiparallel residual blocks (inspired by CenterNet architecture) for 3D localization and poses estimation for vehicles on the scene. The second stage takes the output of the first stage as input (cropped car images) to train EfficientNet B3 for the image recognition task. Using predefined 3D Models, we substitute each vehicle on the scene with its match using the rotation matrix and translation vector from the first stage to get the 3D detection bounding boxes and segmentation masks. We trained our models on an open-source dataset (ApolloCar3D). Our method outperforms all published solutions in terms of 6 degrees of freedom error (6 DoF err).

show abstract

“…To address these problems, many attempts have been made in recent studies. Zhu et al [ 37 ] proposed VPFNet, a network that first generates 3D proposal regions from point clouds and divides them into multiple small grids again, with the corner points of each grid as virtual points, and next projects these points onto the image for image feature sampling and point cloud feature extraction, which reduces the computation and compensates the lack of sparse point clouds. Huang et al [ 20 ] proposed a LI-Fusion module for fusing point cloud and image information at different scales in the feature extraction stage, and then associating the image features to the corresponding point cloud locations, while adding a consistency forcing loss for balancing the impact of classification loss and location loss on the detection results.…”

Section: Related Workmentioning

confidence: 99%

“…In recent years, there are some studies on 3D object detection based on camera and LiDAR fusion, which can be classified into serial type [ 10 , 11 , 12 , 13 , 14 , 15 , 16 ] and parallel type [ 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 ] according to the stage of fusion. The serial type method is represented by F-PointNet [ 13 ] which usually takes the image of the camera as input first and uses image object detection or semantic segmentation algorithm to get the spatial location of the object, then projects it to the LiDAR point cloud to extract the point cloud of the frustum region around the object, and finally uses the normal point cloud 3D object detection algorithm to get 3D bounding boxes.…”

Section: Introductionmentioning

confidence: 99%

Anti-Noise 3D Object Detection of Multimodal Feature Attention Fusion Based on PV-RCNN

Zhu

et al. 2022

Sensors

View full text Add to dashboard Cite

3D object detection methods based on camera and LiDAR fusion are susceptible to environmental noise. Due to the mismatch of physical characteristics of the two sensors, the feature vectors encoded by the feature layer are in different feature spaces. This leads to the problem of feature information deviation, which has an impact on detection performance. To address this problem, a point-guided feature abstract method is presented to fuse the camera and LiDAR at first. The extracted image features and point cloud features are aggregated to keypoints for enhancing information redundancy. Second, the proposed multimodal feature attention (MFA) mechanism is used to achieve adaptive fusion of point cloud features and image features with information from multiple feature spaces. Finally, a projection-based farthest point sampling (P-FPS) is proposed to downsample the raw point cloud, which can project more keypoints onto the close object and improve the sampling rate of the point-guided image features. The 3D bounding boxes of the object is obtained by the region of interest (ROI) pooling layer and the fully connected layer. The proposed 3D object detection algorithm is evaluated on three different datasets, and the proposed algorithm achieved better detection performance and robustness when the image and point cloud data contain rain noise. The test results on a physical test platform further validate the effectiveness of the algorithm.

show abstract

VPFNet: Improving 3D Object Detection With Virtual Point Based LiDAR and Stereo Data Fusion

Cited by 85 publications

References 59 publications

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

Collaborative Perception in Autonomous Driving: Methods, Datasets and Challenges

3D Vehicle Detection and Segmentation Based on EfficientNetB3 and CenterNet Residual Blocks

Anti-Noise 3D Object Detection of Multimodal Feature Attention Fusion Based on PV-RCNN

Contact Info

Product

Resources

About