MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views

Chen, Ke; Oldja, Ryan; Smolyanskiy, Nikolai; Birchfield, Stan; Popov, Alexander; Wehr, David; Eden, Ibrahim; Pehserl, Joachim

doi:10.1109/iros45743.2020.9341450

Cited by 33 publications

(16 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In another approach, 3D-MiniNet [2] proposes a learning-based projection module to extract local and global information from the 3D data and then feeds it to a 2D FCNN in order to generate semantic segmentation predictions. In a slightly different approach, MVLidarNet [7] benefits form range-image LiDAR semantic segmentation to refine object instances in bird's-eye-view perspective, showcasing the applicability of LiDAR semantic segmentation in real-world applications.…”

Section: Hybrid Methodsmentioning

confidence: 99%

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Cheng¹,

Razani²,

Taghavi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Autonomous robotic systems and self driving cars rely on accurate perception of their surroundings as the safety of the passengers and pedestrians is the top priority. Semantic segmentation is one of the essential components of road scene perception that provides semantic information of the surrounding environment. Recently, several methods have been introduced for 3D LiDAR semantic segmentation. While they can lead to improved performance, they are either afflicted by high computational complexity, therefore are inefficient, or they lack fine details of smaller instances. To alleviate these problems, we propose (AF) 2 -S3Net, an end-to-end encoder-decoder CNN network for 3D LiDAR semantic segmentation. We present a novel multibranch attentive feature fusion module in the encoder and a unique adaptive feature selection module with feature map re-weighting in the decoder. Our (AF) 2 -S3Net fuses the voxel-based learning and point-based learning methods into a unified framework to effectively process the large 3D scene. Our experimental results show that the proposed method outperforms the state-of-the-art approaches on the large-scale SemanticKITTI benchmark, ranking 1 st on the competitive public leaderboard competition upon publication.

show abstract

Section: Hybrid Methodsmentioning

confidence: 99%

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Cheng¹,

Razani²,

Taghavi³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A line of works [4,18] realize multi-view fusion either by aggregating features to refine proposals or fusing features in the region constrained by the spatial projection. [7,17] fuse the ROI features from point cloud and camera image for proposals refinement.…”

Section: Multi-view 3d Detectionmentioning

confidence: 99%

VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention

Deng¹,

Liang²,

Sun³

et al. 2022

Preprint

View full text Add to dashboard Cite

Detecting objects from LiDAR point clouds is of tremendous significance in autonomous driving. In spite of good progress, accurate and reliable 3D detection is yet to be achieved due to the sparsity and irregularity of LiDAR point clouds. Among existing strategies, multi-view methods have shown great promise by leveraging the more comprehensive information from both bird's eye view (BEV) and range view (RV). These multi-view methods either refine the proposals predicted from single view via fused features, or fuse the features without considering the global spatial context; their performance is limited consequently. In this paper, we propose to adaptively fuse multi-view features in a global spatial context via Dual Cross-VIew SpaTial Attention (VISTA). The proposed VISTA is a novel plug-and-play fusion module, wherein the multi-layer perceptron widely adopted in standard attention modules is replaced with a convolutional one. Thanks to the learned attention mechanism, VISTA can produce fused features of high quality for prediction of proposals. We decouple the classification and regression tasks in VISTA, and an additional constraint of attention variance is applied that enables the attention module to focus on specific targets instead of generic points. We conduct thorough experiments on the benchmarks of nuScenes and Waymo; results confirm the efficacy of our designs. At the time of submission, our method achieves 63.0% in overall mAP and 69.8% in NDS on the nuScenes benchmark, outperforming all published methods by up to 24% in safety-crucial categories such as cyclist. Code.

show abstract

“…The authors of [ 24 ] extract features from both views and use an early fusion approach. Segmenting the spherical image first and projecting the results into the bird’s-eye view for further processing is done in [ 25 ]. Ref.…”

Section: Related Workmentioning

confidence: 99%

Fused Projection-Based Point Cloud Segmentation

Kellner

Stahl

Reiterer

2022

Sensors

View full text Add to dashboard Cite

Semantic segmentation is used to enable a computer to understand its surrounding environment. In image processing, images are partitioned into segments for this purpose. State-of-the-art methods make use of Convolutional Neural Networks to segment a 2D image. Compared to that, 3D approaches suffer from computational cost and are not applicable without any further steps. In this work, we focus on semantic segmentation based on 3D point clouds. We use the idea to project the 3D data into a 2D image to accelerate the segmentation process. Afterward, the processed image gets re-projected to receive the desired result. We investigate different projection views and compare them to clarify their strengths and weaknesses. To compensate for projection errors and the loss of geometrical information, we evolve the approach and show how to fuse different views. We have decided to fuse the bird’s-eye and the spherical projection as each of them achieves reasonable results, and the two perspectives complement each other best. For training and evaluation, we use the real-world datasets SemanticKITTI. Further, we use the ParisLille and synthetic data generated by the simulation framework Carla to analyze the approaches in more detail and clarify their strengths and weaknesses. Although these methods achieve reasonable and competitive results, they lack flexibility. They depend on the sensor used and the setup in which the sensor is used.

show abstract

MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views

Cited by 33 publications

References 18 publications

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

VISTA: Boosting 3D Object Detection via Dual Cross-VIew SpaTial Attention

Fused Projection-Based Point Cloud Segmentation

Contact Info

Product

Resources

About