Learning SO(3) Equivariant Representations with Spherical CNNs

Abstract

We address the problem of 3D rotation equivariance in convolutional neural networks. 3D rotations have been a challenging nuisance in 3D classification tasks requiring higher capacity and extended data augmentation in order to tackle it. We model 3D data with multivalued spherical functions and we propose a novel spherical convolutional network that implements exact convolutions on the sphere by realizing them in the spherical harmonic domain. Resulting filters have local symmetry and are localized by enforcing smooth spectra. We apply a novel pooling on the spectral domain and our operations are independent of the underlying spherical resolution throughout the network. We show that networks with much lower capacity and without requiring data augmentation can exhibit performance comparable to the state of the art in standard retrieval and classification benchmarks.

Learning an Effective Equivariant 3D Descriptor Without Supervision

“…Hence, the local geometry surrounding a feature point needs to be converted into a spherical representation. A common strategy adopted by [2,7] is to project a 3D mesh onto an enclosing discretized sphere using a raycasting scheme. Since our input data is not a regular watertight mesh, but a point cloud corresponding to the neighborhood of the point we wish to describe, we first convert 3D points into a spherical coordinate system and then construct a quantization grid in this new coordinate system, similarly to [35].…”

“…a potential LRF. This has been already exploited to align full shapes in [7], by finding the arg max of the correlation between two feature maps. Note that we cannot use the same approach in the context of invariant descriptor matching, as this would require a costly computation to compute the distance between every pair of source and target descriptors.…”

“…We argue that the ability to learn an embedding equivariant with respect to rotations of the input is the most sound approach to include pose information in the latent space. To this end, we leverage recent work on Spherical CNNs [2,7], which have enhanced the deep learning machinery by enabling it to learn also rotationequivariant representations from 3D spherical signals by means of correlations defined for the SO(3) group of rotations. Hence, in our architecture, a Spherical CNN encoder learns to summarize the geometry around a feature point into a rotation-equivariant embedding and a decoder warps a 2D grid in order to reconstruct the raw input data.…”

Location Field Descriptors: Single Image 3D Model Retrieval in the Wild

“…A large number of previous works perform retrieval given a query 3D model [48,49]. These methods either directly operate on 3D data, e.g., in the form of voxel grids [41,65], spherical maps [11], or point clouds [42], or process multi-view renderings of the query 3D model [5,10,41,52] to compute a shape descriptor.…”

Representation Learning on Unit Ball with 3D Roto-translational Equivariance

“…As shown in Table 4, our feature vector is able to match the state of the art results achieved by Xie et al (2015). Table 5 depicts the performance comparison on SHREC'17 dataset (as reported in Esteves et al (2018)). This dataset includes random SO(3) perturbations.…”