Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation

Abstract

Online semantic 3D segmentation in company with realtime RGB-D reconstruction poses special challenges such as how to perform 3D convolution directly over the progressively fused 3D geometric data, and how to smartly fuse information from frame to frame. We propose a novel fusionaware 3D point convolution which operates directly on the geometric surface being reconstructed and exploits effectively the inter-frame correlation for high quality 3D feature learning. This is enabled by a dedicated dynamic data structure which organizes the online acquired point cloud with global-local trees. Globally, we compile the online reconstructed 3D points into an incrementally growing coordinate interval tree, enabling fast point insertion and neighborhood query. Locally, we maintain the neighborhood information for each point using an octree whose construction benefits from the fast query of the global tree. Both levels of trees update dynamically and help the 3D convolution effectively exploits the temporal coherence for effective information fusion across RGB-D frames. Through evaluation on public benchmark datasets, we show that our method achieves the state-of-the-art accuracy of semantic segmentation with online RGB-D fusion in 10 FPS.

Multi-Path Region Mining for Weakly Supervised 3D Semantic Segmentation on Point Clouds

“…There are mainly three categories for 3D semantic segmentation methods: projection-based methods, voxel-based methods and point-based methods. Multi-view projection based methods [15,34,8] project the 3D data into 2D from multiple viewpoints, therefore they can easily process the projected data on 2D convolution networks. However, these methods suffer from occlusion, view-point selection, misalignment and other defects that may limit the performance.…”

Semantic Dense Reconstruction with Consistent Scene Segments

“…Semantic segmentation results of 20-class objects/scenarios from different approaches are listed in Table II, FPC [42] achieves good predictions in 5 classes, especially in bath,bed and wall instances. However, it does not understand bookshelves existed in scenes.…”

LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation

“…PointNet [11] is one of the first works of directly learning the point features based on the raw point clouds through a shared Multi-Layer Perceptron (MLP) and max-pooling. Some subsequent works [12], [13], [14], [15], [16], [17], [18], [19], [20] are often based on the pioneering works (e.g., PointNet, PointNet++) and further promote the effectiveness of sampling, grouping and ordering to improve the performance of semantic segmentation. Other methods [21], [22], [23] extract the hierarchical point features by introducing a graph network.…”