RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation

Xu, Jianyun; Zhang, Ruixiang; Dou, Jie; Yushi, Zhu; Sun, Jie; Pu, Shiliang

doi:10.48550/arxiv.2103.12978

Cited by 7 publications

(11 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To save computational cost, regular representations, including 3d voxels and 2d grids, polar and cylinder grids, and range images (Zhou and Tuzel 2018;Zhang et al 2020;Zhu et al 2020b;Milioto et al 2019;Xu et al 2020) are used to organize sparse points. Recently, hybrid methods (Tang et al 2020;Xu et al 2021;Ye et al 2021) that combine multiple representations are proposed to integrate the advantages of both fine-grained point-wise features and effective feature aggregation of regular representations. Sparse convolution (Graham 2015;Graham, Engelcke, and Van Der Maaten 2018) is also widely used to restrict convolution output only in the active regions, accelerating the volumetric convolution and enabling larger model size.…”

Section: Related Workmentioning

confidence: 99%

Sparse Cross-Scale Attention Network for Efficient LiDAR Panoptic Segmentation

Wu²,

et al. 2022

AAAI

View full text Add to dashboard Cite

Two major challenges of 3D LiDAR Panoptic Segmentation (PS) are that point clouds of an object are surface-aggregated and thus hard to model the long-range dependency especially for large instances, and that objects are too close to separate each other. Recent literature addresses these problems by time-consuming grouping processes such as dual-clustering, mean-shift offsets and etc., or by bird-eye-view (BEV) dense centroid representation that downplays geometry. However, the long-range geometry relationship has not been sufficiently modeled by local feature learning from the above methods. To this end, we present SCAN, a novel sparse cross-scale attention network to first align multi-scale sparse features with global voxel-encoded attention to capture the long-range relationship of instance context, which is able to boost the regression accuracy of the over-segmented large objects. For the surface-aggregated points, SCAN adopts a novel sparse class-agnostic representation of instance centroids, which can not only maintain the sparsity of aligned features to solve the under-segmentation on small objects, but also reduce the computation amount of the network through sparse convolution. Our method outperforms previous methods by a large margin in the SemanticKITTI dataset for the challenging 3D PS task, achieving 1st place with a real-time inference speed.

show abstract

Section: Related Workmentioning

confidence: 99%

Sparse Cross-Scale Attention Network for Efficient LiDAR Panoptic Segmentation

Wu²,

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Multi-view fusion-based methods combine voxel-based, projection-based and/or point-wise operations for LiDAR point clouds segmentation. To extract more semantic information, some recent methods [35], [36], [37], [38], [39], [40], [41], [7], [8] blend two or more different views together. For instance, [38], [39] combine point-wise information from BEV and range-image in early-stage, and then feed it to the subsequent network.…”

Section: B Lidar Point Cloud Semantic Segmentationmentioning

confidence: 99%

“…2 -S3Net [7] use point-voxel fusion scheme to achieve better segmentation results. RPVNet [8] proposes a deep fusion network to fuse range-point-voxel three views by a gated fusion mechanism. However, the performance of these methods is also limited due to the LiDAR point clouds lacking rich colors and textures.…”

Section: B Lidar Point Cloud Semantic Segmentationmentioning

confidence: 99%

“…As mentioned above, although LiDAR points can provide accurate distance measurements and capture the structures of objects, they are usually sparse, unordered, and unevenly distributed. Recently, some methods [5], [6], [7], [8] based only on LiDAR have significantly improved the performance of 3D semantic segmentation, but the performance of these methods are still limited because of lacking dense and rich information on the objects such as their colors, and textures, as depicted in the right of Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

Zhao,

Zhou,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

Camera and 3D LiDAR sensors have become indispensable devices in modern autonomous driving vehicles, where the camera provides the fine-grained texture, color information in 2D space and LiDAR captures more precise and farther-away distance measurements of the surrounding environments. The complementary information from these two sensors makes the two-modality fusion be a desired option. However, two major issues of the fusion between camera and LiDAR hinder its performance, i.e., how to effectively fuse these two modalities and how to precisely align them (suffering from the weak spatiotemporal synchronization problem). In this paper, we propose a coarse-to-fine LiDAR and camera fusion-based network (termed as LIF-Seg) for LiDAR segmentation. For the first issue, unlike these previous works fusing the point cloud and image information in a one-to-one manner, the proposed method fully utilizes the contextual information of images and introduces a simple but effective early-fusion strategy. Second, due to the weak spatiotemporal synchronization problem, an offset rectification approach is designed to align these two-modality features. The cooperation of these two components leads to the success of the effective camera-LiDAR fusion. Experimental results on the nuScenes dataset show the superiority of the proposed LIF-Seg over existing methods with a large margin. Ablation studies and analyses demonstrate that our proposed LIF-Seg can effectively tackle the weak spatiotemporal synchronization problem.

show abstract

“…Cylinder3D is a novelty 3D segmentation approach that uses cylindrical and asymmetrical 3D convolution networks for achieving SOTA performance. In addition, any SOTA 3D segmentators such as Polarnet [37], AF2S3Net [38], RPVNet [39] can be used for the proposed framework. Furthermore, any extra pointwise semantic annotations are not required, we can freely use the 3D object bounding boxes annotations for generating the segmentation labels.…”

Section: A Multi-modal Semantic Segmentationmentioning

confidence: 99%

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

Xü

Zhou

Yin

et al. 2021

Preprint

View full text Add to dashboard Cite

Accurate detection of obstacles in 3D is an essential task for autonomous driving and intelligent transportation. In this work, we propose a general multimodal fusion framework FusionPainting to fuse the 2D RGB image and 3D point clouds at a semantic level for boosting the 3D object detection task. Especially, the FusionPainting framework consists of three main modules: a multi-modal semantic segmentation module, an adaptive attention-based semantic fusion module, and a 3D object detector. First, semantic information is obtained for 2D image and 3D Lidar point clouds based on 2D and 3D segmentation approaches. Then the segmentation results from different sensors are adaptively fused based on the proposed attention-based semantic fusion module. Finally, the point clouds painted with the fused semantic label are sent to the 3D detector for obtaining the 3D objection results. The effectiveness of the proposed framework has been verified on the large-scale nuScenes detection benchmark by comparing with three different baselines. The experimental results show that the fusion strategy can significantly improve the detection performance compared to the methods using only point clouds, and the methods using point clouds only painted with 2D segmentation information. Furthermore, the proposed approach outperforms other state-of-the-art methods on the nuScenes testing benchmark.

show abstract

RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation

Cited by 7 publications

References 0 publications

Sparse Cross-Scale Attention Network for Efficient LiDAR Panoptic Segmentation

Sparse Cross-Scale Attention Network for Efficient LiDAR Panoptic Segmentation

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

Contact Info

Product

Resources

About