SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor

Xu, Jiachen; Gong, Jingyu; Zhou, Jie; Tan, Xin; Xie, Yuan; Ma, Lizhuang

doi:10.24963/ijcai.2020/84

Cited by 18 publications

(20 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Meanwhile, the graph-based method [ 75 , 76 ] uses a graph convolutional operation for point clouds. In addition, some recent studies have shown the importance of global contexts when applying these methods to semantic segmentation tasks [ 77 , 78 , 79 ]. The second limitation is that we cannot handle waveform specific features.…”

Section: Resultsmentioning

confidence: 99%

FWNet: Semantic Segmentation for Full-Waveform LiDAR Data Using Deep Learning

Shinohara

Xiu

Matsuoka

2020

Sensors

View full text Add to dashboard Cite

In the computer vision field, many 3D deep learning models that directly manage 3D point clouds (proposed after PointNet) have been published. Moreover, deep learning-based-techniques have demonstrated state-of-the-art performance for supervised learning tasks on 3D point cloud data, such as classification and segmentation tasks for open datasets in competitions. Furthermore, many researchers have attempted to apply these deep learning-based techniques to 3D point clouds observed by aerial laser scanners (ALSs). However, most of these studies were developed for 3D point clouds without radiometric information. In this paper, we investigate the possibility of using a deep learning method to solve the semantic segmentation task of airborne full-waveform light detection and ranging (lidar) data that consists of geometric information and radiometric waveform data. Thus, we propose a data-driven semantic segmentation model called the full-waveform network (FWNet), which handles the waveform of full-waveform lidar data without any conversion process, such as projection onto a 2D grid or calculating handcrafted features. Our FWNet is based on a PointNet-based architecture, which can extract the local and global features of each input waveform data, along with its corresponding geographical coordinates. Subsequently, the classifier consists of 1D convolutional operational layers, which predict the class vector corresponding to the input waveform from the extracted local and global features. Our trained FWNet achieved higher scores in its recall, precision, and F1 score for unseen test data—higher scores than those of previously proposed methods in full-waveform lidar data analysis domain. Specifically, our FWNet achieved a mean recall of 0.73, a mean precision of 0.81, and a mean F1 score of 0.76. We further performed an ablation study, that is assessing the effectiveness of our proposed method, of the above-mentioned metric. Moreover, we investigated the effectiveness of our PointNet based local and global feature extraction method using the visualization of the feature vector. In this way, we have shown that our network for local and global feature extraction allows training with semantic segmentation without requiring expert knowledge on full-waveform lidar data or translation into 2D images or voxels.

show abstract

Section: Resultsmentioning

confidence: 99%

FWNet: Semantic Segmentation for Full-Waveform LiDAR Data Using Deep Learning

Shinohara

Xiu

Matsuoka

2020

Sensors

View full text Add to dashboard Cite

show abstract

“…HPEIN [20] extracted features of edges between neighboring points to implicitly model the neighborhood relation. Region similarity loss was proposed to propagate distinguishing features of center points to neighbors with the same categories in a local neighborhood [2].…”

Section: Neighborhood Context Learningmentioning

confidence: 99%

“…As a basic form of 3D data, the point cloud is very popular and can be easily converted into meshes or voxels [1]. Semantic segmentation of point clouds is an essential 3D scene comprehension task yet remains challenging due to its inherent irregularity [2].…”

Section: Introductionmentioning

confidence: 99%

Neighborhood co-occurrence modeling in 3D point cloud segmentation

Gong

Zhou

2021

Comp. Visual Media

Self Cite

View full text Add to dashboard Cite

A significant performance boost has been achieved in point cloud semantic segmentation by utilization of the encoder-decoder architecture and novel convolution operations for point clouds. However, co-occurrence relationships within a local region which can directly influence segmentation results are usually ignored by current works. In this paper, we propose a neighborhood co-occurrence matrix (NCM) to model local co-occurrence relationships in a point cloud. We generate target NCM and prediction NCM from semantic labels and a prediction map respectively. Then, Kullback-Leibler (KL) divergence is used to maximize the similarity between the target and prediction NCMs to learn the co-occurrence relationship. Moreover, for large scenes where the NCMs for a sampled point cloud and the whole scene differ greatly, we introduce a reverse form of KL divergence which can better handle the difference to supervise the prediction NCMs. We integrate our method into an existing backbone and conduct comprehensive experiments on three datasets: Semantic3D for outdoor space segmentation, and S3DIS and ScanNet v2 for indoor scene segmentation. Results indicate that our method can significantly improve upon the backbone and outperform many leading competitors.

show abstract

“…Additionally, the common up-sampling methods using nearest neighbors cannot trace the encoding relationship, thus introducing improper supervisions to the intermediate features (referring Sec 4.4 for discussion). More recently, SceneEncoder [37] provided a method to supervise the center-most layer to extract meaningful global features, but lots of other layers remain unhandled.…”

Section: Introductionmentioning

confidence: 99%

“…Inspired by SceneEncoder [37], for each sampled point in any layer of encoder, according to the existence of categories in its receptive field, a multi-hot binary code can be built, designated as target Receptive Field Component Code (RFCC). The target RFCCs at different layers are gen-erated alongside the convolution and down-sampling, thus they can precisely record the existing categories in corresponding receptive fields without any extra annotations.…”

Section: Introductionmentioning

confidence: 99%

Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning

Gong¹,

Xu²,

Tan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Hidden features in neural network usually fail to learn informative representation for 3D segmentation as supervisions are only given on output prediction, while this can be solved by omni-scale supervision on intermediate layers. In this paper, we bring the first omni-scale supervision method to point cloud segmentation via the proposed gradual Receptive Field Component Reasoning (RFCR), where target Receptive Field Component Codes (RFCCs) are designed to record categories within receptive fields for hidden units in the encoder. Then, target RFCCs will supervise the decoder to gradually infer the RFCCs in a coarse-tofine categories reasoning manner, and finally obtain the semantic labels. Because many hidden features are inactive with tiny magnitude and make minor contributions to RFCC prediction, we propose a Feature Densification with a centrifugal potential to obtain more unambiguous features, and it is in effect equivalent to entropy regularization over features. More active features can further unleash the potential of our omni-supervision method. We embed our method into four prevailing backbones and test on three challenging benchmarks. Our method can significantly improve the backbones in all three datasets. Specifically, our method brings new state-of-the-art performances for S3DIS as well as Semantic3D and ranks the 1st in the ScanNet benchmark among all the point-based methods. Code will be publicly available at https://github.com/azuki-miho/RFCR.

show abstract

SceneEncoder: Scene-Aware Semantic Segmentation of Point Clouds with A Learnable Scene Descriptor

Cited by 18 publications

References 1 publication

FWNet: Semantic Segmentation for Full-Waveform LiDAR Data Using Deep Learning

FWNet: Semantic Segmentation for Full-Waveform LiDAR Data Using Deep Learning

Neighborhood co-occurrence modeling in 3D point cloud segmentation

Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning

Contact Info

Product

Resources

About