Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

Soltani, Amir; Huang, Haibin; Wu, Jiajun; Kulkarni, Tejas; Tenenbaum, Joshua B.

doi:10.1109/cvpr.2017.269

Cited by 182 publications

(152 citation statements)

References 20 publications

Supporting

Mentioning

152

Contrasting

Order By: Relevance

“…Most of the missclassification comes from the similar category pairs like (table, desk ) and (night stand, dresser ). We also perform better than all the appearance based methods, except Wang et al [27] which performs a specialized view based ModelNet40 ModelNet10 T converge # views a Data (ModelNet40) # aug inst b aug Image based PANORAMA NN [25] 91.12 90.70 30 mins 1 a N Wang et al [27] 93.80 -20 hours 12 a N Pairwise [8] 90.70 92.80 -12 a N MVCNN [28] 90.10 --80 a N Geometry based Kd-Net depth 10 [9] 90.60 93. 30 clustering for the task of classification and takes 10 times longer to converge than our algorithm.…”

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 79%

Learning 3D Shapes as Multi-layered Height-Maps Using 2D Convolutional Networks

Sarkar

Hampiholi

Varanasi

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

We present a novel global representation of 3D shapes, suitable for the application of 2D CNNs. We represent 3D shapes as multilayered height-maps (MLH) where at each grid location, we store multiple instances of height maps, thereby representing 3D shape detail that is hidden behind several layers of occlusion. We provide a novel view merging method for combining view dependent information (Eg. MLH descriptors) from multiple views. Because of the ability of using 2D CNNs, our method is highly memory efficient in terms of input resolution compared to the voxel based input. Together with MLH descriptors and our multi view merging, we achieve the state-of-the-art result in classification on ModelNet dataset.

show abstract

Section: Comparison With the State-of-the-art Methodsmentioning

confidence: 79%

Learning 3D Shapes as Multi-layered Height-Maps Using 2D Convolutional Networks

Sarkar

Hampiholi

Varanasi

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

show abstract

“…The unordered point generation method (Fan, Su, and Guibas 2017) gets sparse point clouds which are limited to characterizing enough details, leading to large chamfer distances. The method proposed by Soltani et al(Soltani et al 2017) also obtains small distances since it generates points with many more (20) depth maps.…”

Section: Reconstruction Resultsmentioning

confidence: 99%

MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image

Wang

Sun

2019

AAAI

View full text Add to dashboard Cite

In this paper, we address the problem of reconstructing an object's surface from a single image using generative networks. First, we represent a 3D surface with an aggregation of dense point clouds from multiple views. Each point cloud is embedded in a regular 2D grid aligned on an image plane of a viewpoint, making the point cloud convolution-favored and ordered so as to fit into deep network architectures. The point clouds can be easily triangulated by exploiting connectivities of the 2D grids to form mesh-based surfaces. Second, we propose an encoder-decoder network that generates such kind of multiple view-dependent point clouds from a single image by regressing their 3D coordinates and visibilities. We also introduce a novel geometric loss that is able to interpret discrepancy over 3D surfaces as opposed to 2D projective planes, resorting to the surface discretization on the constructed meshes. We demonstrate that the multi-view point regression network outperforms state-of-the-art methods with a significant improvement on challenging datasets.

show abstract

“…This architecture has been used to learn a continuous latent space for volumetric [17], [18], depthbaed [19], surface-based [20], and point-based [21] 3D reconstruction. In Wu et al [17], for example, the image encoder takes as input a 256×256 RGB image and outputs two 200-dimensional vectors representing, respectively, the mean and the standard deviation of a Gaussian distribution in the 200-dimensional space.…”

Section: Continuous Latent Spacesmentioning

confidence: 99%

“…The approach uses a standard encoder-decoder and an additional network composed of three fully-connected layers to encode the viewpoint. Soltani et al [19] and Lin et al [69] followed the same approach but predicts the depth maps, along with their binary masks, from predefined view points. In both methods, the merging is performed in a post-processing step.…”

Section: Intermediatingmentioning

confidence: 99%

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Han

Laga

Bennamoun

2021

IEEE Trans. Pattern Anal. Mach. Intell.

320

150

View full text Add to dashboard Cite

3D reconstruction is a longstanding ill-posed problem, which has been explored for decades by the computer vision, computer graphics, and machine learning communities. Since 2015, image-based 3D reconstruction using convolutional neural networks (CNN) has attracted increasing interest and demonstrated an impressive performance. Given this new era of rapid evolution, this article provides a comprehensive survey of the recent developments in this field. We focus on the works which use deep learning techniques to estimate the 3D shape of generic objects either from a single or multiple RGB images. We organize the literature based on the shape representations, the network architectures, and the training mechanisms they use. While this survey is intended for methods which reconstruct generic objects, we also review some of the recent works which focus on specific object classes such as human body shapes and faces. We provide an analysis and comparison of the performance of some key papers, summarize some of the open problems in this field, and discuss promising directions for future research.

show abstract

Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

Abstract: Our method generates multi-view depth maps and silhouettes, and uses a rendering function to obtain the 3D shapes. Right: We can also extend our framework to reconstruct 3D shapes from single/multi-view depth maps or silhouettes.

Cited by 182 publications

References 20 publications

Learning 3D Shapes as Multi-layered Height-Maps Using 2D Convolutional Networks

Learning 3D Shapes as Multi-layered Height-Maps Using 2D Convolutional Networks

MVPNet: Multi-View Point Regression Networks for 3D Object Reconstruction from A Single Image

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Contact Info

Product

Resources

About