VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification

Abstract

Multi-view deep neural network is perhaps the most successful approach in 3D shape classification. However, the fusion of multi-view features based on max or average pooling lacks a view selection mechanism, limiting its application in, e.g., multi-view active object recognition by a robot. This paper presents VERAM, a recurrent attention model capable of actively selecting a sequence of views for highly accurate 3D shape classification. VERAM addresses an important issue commonly found in existing attention-based models, i.e., the unbalanced training of the subnetworks corresponding to next view estimation and shape classification. The classification subnetwork is easily overfitted while the view estimation one is usually poorly trained, leading to a suboptimal classification performance. This is surmounted by three essential view-enhancement strategies: 1) enhancing the information flow of gradient backpropagation for the view estimation subnetwork, 2) devising a highly informative reward function for the reinforcement training of view estimation and 3) formulating a novel loss function that explicitly circumvents view duplication. Taking grayscale image as input and AlexNet as CNN architecture, VERAM with 9 views achieves instance-level and class-level accuracy of 95.5% and 95.3% on ModelNet10, 93.7% and 92.1% on ModelNet40, both are the state-of-the-art performance under the same number of views.

Recurrent 3D attentional networks for end-to-end active object recognition

“…Different from their method, our attention model is trained offline hence no online sampling is required, making it efficient for online active recognition. The works of Jayaraman and Grauman [11], Xu et al [32] and Chen et al [5] are the most related to ours. Compared to MV-RNN [32], VERAM [5] explicitly integrates view confidence and view location constrains into reward function, and deploy some strategies to do view-enhancement.…”

“…We also notice that VERAM [5] achieves recognition accuracy of 92.1% on ModelNet40 with 9 views of gray images. They align all shapes and render 144 gray images for each shape with Phong reflection model.…”

“…Some works formulate active recognition as a reinforcement learning problem, to learn a viewing policy under various observations. Especially, a few recent works attempted end-to-end reinforcement learning based on recurrent neural networks [5,11,32]. Applying a learned policy is apparently much more efficient than sampling from a generative model.…”

Semi‐Supervised 3D Shape Recognition via Multimodal Deep Co‐training

“…Due to the appearance of large labeled 3D shape repository, such as ModelNet [WSK∗15] and ShapeNet [CFG∗15], supervised deep learning becomes the mainstreaming technology for 3D shape feature learning. With the class labels, informative features can be produced by constructing the mapping between the class labels and different 3D raw representations, such as voxels [WSK∗15], meshes [FFY∗19], points clouds [QSMG16, QYSG17, LFXP19, WSL∗19] and views [SMKLM15, CZZ∗18]. Compared with the unsupervised methods, these methods achieve high classification accuracy.…”

View N-Gram Network for 3D Object Retrieval

“…This holds for both aligned and unaligned 3D shapes. However, RNN based methods may be sensitive to the predefined viewpoints and thus cannot ensure the rotational invariance of the learned features, as suggested in [11].…”