Occupancy Networks: Learning 3D Reconstruction in Function Space

Abstract

With the advent of deep neural networks, learning-based approaches for 3D reconstruction have gained popularity. However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometry of arbitrary topology. Many of the state-of-the-art learningbased 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricted domain. In this paper, we propose Occupancy Networks, a new representation for learning-based 3D reconstruction methods. Occupancy networks implicitly represent the 3D surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches, our representation encodes a description of the 3D output at infinite resolution without excessive memory footprint. We validate that our representation can efficiently encode 3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both qualitatively and quantitatively, for the challenging tasks of 3D reconstruction from single images, noisy point clouds and coarse discrete voxel grids. We believe that occupancy networks will become a useful tool in a wide variety of learning-based 3D tasks.

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

“…This approach shows its strength in reconstructing high-fidelity 3D geometry without having to keep a discretized representation of the entire output volume in memory simultaneously. Furthermore, unlike implicit surface representations using a global feature vector [29,32,10], PIFu utilizes fully convolutional image features, retaining local details present in an input image.…”

Learning Shape Templates With Structured Implicit Functions

“…We show that this DSIF representation improves both reconstruction accuracy and generalization behavior over previous work -its F-Score results are better than the stateof-the-art [21] by 10.3 points for 3D autoencoding of test models from trained classes and by 17.8 points for unseen classes. We show that it dramatically reduces network parameter count -its local decoder requires approximately 0.4% of the parameters used by [21]. We show that it can be used to complete posed depth images -its depth completion results are 15.8 percentage points higher than [21].…”

“…sum of local 3D functions, each evaluated as the product of a Gaussian and a residual function predicted with a deep network. We describe a method for inferring a DSIF from a 3D surface or posed depth image by first predicting a structured decomposition into shape elements, encoding 3D points within each shape element using PointNet [27], and decoding them with a residual TinyOccNet [21]. This approach provides an end-to-end framework for encoding shapes in local regions arranged in a global structure.…”

“…The deep implicit function f defines local shape details within a shape element by modulating g (one f function is shared by all shape elements). To compute f , we use a network architecture based on Occupancy Networks [21] (we call ours TinyOccNet). As in the original OccNet, ours is organized as a fully-connected network conditioned on the latent code z i and trained using conditional batch normalization.…”

360-Degree Textures of People in Clothing from a Single Image

“…skirts or dresses. Implicit function based representations [71,56,46,48,31] might be beneficial to deal with different topologies, but they do not allow control. Although it is remarkable that our model can predict the occluded appearance of the person, the model struggles to predict high frequency detail and complex texture patterns.…”