PointConv: Deep Convolutional Networks on 3D Point Clouds

Abstract

Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and density functions through kernel density estimation. The most important contribution of this work is a novel reformulation proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure.

Multi-View PointNet for 3D Scene Understanding

“…1 shows our performance compared to the published state-of-the-art point cloud based methods on the test set. Our MVPNet outperforms all the published point cloud based methods, like PointConv [28] and PointCNN [14], by a large margin. This confirms the effectiveness of our approach of elevating 2D image features to 3D for geometric fusion, especially for classes with flat shapes, i.e.…”

“…3D Networks CNNs are the state-of-the-art on 2D RGB images, but competing network families exist for 3D data: 3D CNNs [17,11] make use of the voxel representation where the raw point cloud data is transformed into a discrete grid of cells and in practice most of the cells are empty and only voxels that lie on the object surface are occupied. On the other hand, point cloud based networks [19,12,27,14,28,29] can directly take point clouds as input. In our work, we use point cloud based networks, because of their inherent sparsity as compared to voxelbased methods.…”

“…However, not all objects can be distinguished by their shape, especially when they have flat surfaces such as doors, refrigerators and curtains. Therefore, additional color information should be leveraged, but recent results [28] have shown that naively feeding colored point cloud (XYZRGB) to point cloud based networks does only marginally improve the performance over simple point cloud input (XYZ).…”

Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation

“…PointCNN [7] explores convolution on point clouds and addresses the point ordering issue by permuting and weighting input points and features with the X -Conv operator. Besides, methods of [16,22,27,26,24] explore local context based on graphs.…”

Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds

“…Nevertheless, these primitive GCNs are yet to be seen as a viable solution for point cloud processing due to their inability to effectively handle real-world point clouds. Based on the convolution operation, GCNs can be divided into two groups, namely; the spectral networks [24], [25], [26], [27] and the spatial networks [28], [29], [30], [31], [32]. The former perform convolutions using the graph Laplacian and adjacency matrices, whereas the latter perform convolutions directly in the spatial domain.…”