Deep Projective 3D Semantic Segmentation

Abstract

Semantic segmentation of 3D point clouds is a challenging problem with numerous real-world applications. While deep learning has revolutionized the field of image semantic segmentation, its impact on point cloud data has been limited so far. Recent attempts, based on 3D deep learning approaches (3D-CNNs), have achieved below-expected results. Such methods require voxelizations of the underlying point cloud data, leading to decreased spatial resolution and increased memory consumption. Additionally, 3D-CNNs greatly suffer from the limited availability of annotated datasets. In this paper, we propose an alternative framework that avoids the limitations of 3D-CNNs. Instead of directly solving the problem in 3D, we first project the point cloud onto a set of synthetic 2D-images. These images are then used as input to a 2D-CNN, designed for semantic segmentation. Finally, the obtained prediction scores are re-projected to the point cloud to obtain the segmentation results. We further investigate the impact of multiple modalities, such as color, depth and surface normals, in a multi-stream network architecture. Experiments are performed on the recent Semantic3D dataset. Our approach sets a new stateof-the-art by achieving a relative gain of 7.9%, compared to the previous best approach.

Indoor point cloud semantic segmentation based on direction perception and hole sampling

“…Su et al (2015) obtained 2D images of point clouds from different viewpoints by taking 3D point cloud data from different angles in multiple directions and later using a CNN that processes the images to extract features to classify and segment the point clouds. Lawin et al (2017) projected point clouds from different perspectives onto a set of 2D images, and completed semantic segmentation of point clouds by predicting the score of each pixel in the image. Tatarchenko et al (2018) proposed a tangent convolution-based u-shaped network architecture for the point cloud semantic segmentation.…”