Learning Category-Specific Deformable 3D Models for Object Reconstruction

Abstract

We address the problem of fully automatic object localization and reconstruction from a single image. This is both a very challenging and very important problem which has, until recently, received limited attention due to difficulties in segmenting objects and predicting their poses. Here we leverage recent advances in learning convolutional networks for object detection and segmentation and introduce a complementary network for the task of camera viewpoint prediction. These predictors are very powerful, but still not perfect given the stringent requirements of shape reconstruction. Our main contribution is a new class of deformable 3D models that can be robustly fitted to images based on noisy pose and silhouette estimates computed upstream and that can be learned directly from 2D annotations available in object detection datasets. Our models capture top-down information about the main global modes of shape variation within a class providing a "low-frequency" shape. In order to capture fine instance-specific shape details, we fuse it with a high-frequency component recovered from shading cues. A comprehensive quantitative analysis and ablation study on the PASCAL 3D+ dataset validates the approach as we show fully automatic reconstructions on PASCAL VOC as well as large improvements on the task of viewpoint prediction.

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

“…PASCAL3D+ [39] Similar to [16,35], we evaluate our method on the the PASCAL3D+ dataset which consists of PASCAL VOC and ImageNet images for 12 rigid object categories with a set of sparse keypoints annotated on each…”

Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation

“…Alternatively, 3D shape can be also generated by deforming an initialization, which is more related to our work. Tulsiani et al [43] and Kanazawa et al [19] learn a category-specific 3D deformable model and reasons the shape deformations in different images. Wang et al [45] learn to deform an initial ellipsoid to the desired shape in a coarse to fine fashion.…”

3D computer vision based on machine learning with deep neural networks: A review

“…Later work attempts to predict full 3D models as opposed to just depth maps using voxel mappings and joint classification, multiple 2D views of an object, and using an ensemble of complimentary schemes …”

“…Later work attempts to predict full 3D models as opposed to just depth maps using voxel mappings and joint classification, 39 multiple 2D views of an object, 48,49 and using an ensemble of complimentary schemes. 50 As discussed in the last section, Wu et al 39 develop a voxel prediction network for the purpose of 3D model classification given a single RGB-D image of the model. The proposed network is designed for classification and uses randomization (through Gibbs sampling) to predict the occluded points as free space or filled.…”