Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey of Datasets and Methods

Gao, Biao; Pan, Yancheng; Li, Chengkun; Geng, Sibo; Zhao, Huijing

doi:10.1109/tits.2021.3076844

Cited by 62 publications

(35 citation statements)

References 153 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Point-based networks [64,70,99,100,128,139] remedied this problem by operating on points and quickly became popular for segmentation, though generative task like SSC are still a challenge. We refer readers to dedicated surveys on semantic segmentation [38,148].…”

Section: Historical Backgroundmentioning

confidence: 99%

See 1 more Smart Citation

3D Semantic Scene Completion: a Survey

Roldão¹,

Charette²,

Verroust-Blondet³

2021

Preprint

View full text Add to dashboard Cite

Dataset Year Type Nature Input →Ground truth 3D Sensor # Classes Tasks* #Sequences #Frames SSC DE SPE SS OC SNE NYUv2 [115] a 2012 Real-world † Indoor RGB-D → Mesh/Voxel RGB-D 894 (11) 1449 795/654 SUN3D [146] 2013 Real-world Indoor RGB-D → Points RGB-D -254 -NYUCAD [34] b 2013 Synthetic Indoor RGB-D → Mesh/Voxel RGB-D 894 (11) 1449 795/654 SceneNet [55] 2015 Synthetic Indoor RGB-D → Mesh RGB-D ‡ 11 57 -SceneNN [57] 2016 Real-world Indoor RGB-D → Mesh RGB-D 22 100 -SUNCG [118] 2017 Synthetic Indoor Depth → Mesh/Voxel RGB-D ‡ 84 (11) 45622 139368/470 Matterport3D [10] 2017 Real-world Indoor RGB-D → Mesh 3D Scanner 40 (11) 707 72/18 ScanNet [21] 2017 Real-world Indoor RGB-D → Mesh RGB-D 20 (11) 1513 1201/312 2D-3D-S [3] 2017 Real-world Indoor RGB-D → Mesh 3D Scanner 13 270 -SUNCG-RGBD [76] c 2018 Synthetic Indoor RGB-D → Mesh/Voxel RGB-D ‡ 84 (11) 45622 13011/499 SemanticKITTI [4] 2019 Real-world Outdoor Points/RGB → Points/Voxel Lidar-64 28 (19) 22 23201/20351 SynthCity [44] 2019 Synthetic Outdoor Points → Points Lidar ‡ 9 9 -CompleteScanNet [145] d 2020 Real-world † Indoor RGB-D → Mesh/Voxel RGB-D 11 1513 45448/11238 SemanticPOSS [91] 2020 Real-world Outdoor Points/RGB → Points Lidar-40 14 2988 - ‡ Simulated sensor. † Synthetically augmented. a Mesh annotations from [47]. b Derivates from NYUv2 [115] by rendering depth images from mesh annotation. c Derivates from subset of SUNCG [118] where missing RGB images were rendered. d Derivates from ScanNet [21] by fitting CAD models to dense mesh.

show abstract

Section: Historical Backgroundmentioning

confidence: 99%

“…We also encourage the use of autonomous driving simulators such as CARLA [27], SYNTHIA [110] for synthetic dataset generation, devoid of dynamic objects and subsequent registration problems. More extensive surveys on RGB-D and Lidar datasets are provided in [33,38].…”

Section: Semantic Class Balancingmentioning

confidence: 99%

3D Semantic Scene Completion: a Survey

Roldão¹,

Charette²,

Verroust-Blondet³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To solve the problem of semantic segmentation of 3D point cloud data, three types of deep learning methods have been roughly formed, i.e., point-based methods, imagebased methods, and voxel-based methods [3]. The point-based methods use points as the input data of the CNNs, where the difficulty lies in how to extract the feature information of the points.…”

Section: Related Work 21 Point Cloud Semantic Segmentationmentioning

confidence: 99%

LiDAR-Based SLAM under Semantic Constraints in Dynamic Environments

et al. 2021

View full text Add to dashboard Cite

Facing the realistic demands of the application environment of robots, the application of simultaneous localisation and mapping (SLAM) has gradually moved from static environments to complex dynamic environments, while traditional SLAM methods usually result in pose estimation deviations caused by errors in data association due to the interference of dynamic elements in the environment. This problem is effectively solved in the present study by proposing a SLAM approach based on light detection and ranging (LiDAR) under semantic constraints in dynamic environments. Four main modules are used for the projection of point cloud data, semantic segmentation, dynamic element screening, and semantic map construction. A LiDAR point cloud semantic segmentation network SANet based on a spatial attention mechanism is proposed, which significantly improves the real-time performance and accuracy of point cloud semantic segmentation. A dynamic element selection algorithm is designed and used with prior knowledge to significantly reduce the pose estimation deviations caused by SLAM dynamic elements. The results of experiments conducted on the public datasets SemanticKITTI, KITTI, and SemanticPOSS show that the accuracy and robustness of the proposed approach are significantly improved.

show abstract

“…The performance limitation caused by insufficient training data is called "the data hungry effect" [8]. As described in [9], 3DSS studies that use deep learning techniques suffer severe This work was supported in part by the NSFC under Grant 61973004. Yancheng Pan and Huijing Zhao are with the Key Laboratory of Machine Perception (MOE), School of AI, Peking University, Beijing 100084, China (e-mail: panyancheng@pku.edu.cn; zhaohj@pku.edu.cn).…”

Section: Introductionmentioning

confidence: 99%

Understanding the Challenges When 3D Semantic Segmentation Faces Class Imbalanced and OOD Data

Pan¹,

Xie²,

Hong-yan³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

3D semantic segmentation (3DSS) is an essential process in the creation of a safe autonomous driving system. However, deep learning models for 3D semantic segmentation often suffer from the class imbalance problem and out-ofdistribution (OOD) data. In this study, we explore how the class imbalance problem affects 3DSS performance and whether the model can detect the category prediction correctness, or whether data is ID (in-distribution) or OOD. For these purposes, we conduct two experiments using three representative 3DSS models and five trust scoring methods, and conduct both a confusion and feature analysis of each class. Furthermore, a data augmentation method for the 3D LiDAR dataset is proposed to create a new dataset based on SemanticKITTI and SemanticPOSS, called AugKITTI. We propose the wPre metric and TSD for a more in-depth analysis of the results, and follow are proposals with an insightful discussion. Based on the experimental results, we find that: (1) the classes are not only imbalanced in their data size but also in the basic properties of each semantic category.(2) The intraclass diversity and interclass ambiguity make class learning difficult and greatly limit the models' performance, creating the challenges of semantic and data gaps. (3) The trust scores are unreliable for classes whose features are confused with other classes. For 3DSS models, those misclassified ID classes and OODs may also be given high trust scores, making the 3DSS predictions unreliable, and leading to the challenges in judging 3DSS result trustworthiness. All of these outcomes point to several research directions for improving the performance and reliability of the 3DSS models used for real-world applications.

show abstract

Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey of Datasets and Methods

Cited by 62 publications

References 153 publications

3D Semantic Scene Completion: a Survey

3D Semantic Scene Completion: a Survey

LiDAR-Based SLAM under Semantic Constraints in Dynamic Environments

Understanding the Challenges When 3D Semantic Segmentation Faces Class Imbalanced and OOD Data

Contact Info

Product

Resources

About