Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

Abstract

We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image. Limited by the nature of deep neural network, previous methods usually represent a 3D shape in volume or point cloud, and it is non-trivial to convert them to the more ready-to-use mesh model. Unlike the existing methods, our network represents 3D mesh in a graph-based convolutional neural network and produces correct geometry by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image. We adopt a coarse-to-fine strategy to make the whole deformation procedure stable, and define various of mesh related losses to capture properties of different levels to guarantee visually appealing and physically accurate 3D geometry. Extensive experiments show that our method not only qualitatively produces mesh model with better details, but also achieves higher 3D shape estimation accuracy compared to the state-of-the-art.

GraphX-Convolution for Point Cloud Deformation in 2D-to-3D Conversion

“…Tulsianni et al [29] introduced ray-tracing into the picture to predict multiple semantics from an image including a 3D voxel model. Howbeit, voxel representation is known to be inefficient and computationally unfriendly [4,30]. For mesh representation, Wang et al [30] gradually deformed an elliptical mesh given an input image by using graph convolution, but mesh representation requires overhead construction, and graph convolution may result in computing redundancy as masking is needed.…”

“…Howbeit, voxel representation is known to be inefficient and computationally unfriendly [4,30]. For mesh representation, Wang et al [30] gradually deformed an elliptical mesh given an input image by using graph convolution, but mesh representation requires overhead construction, and graph convolution may result in computing redundancy as masking is needed. There has been a number of studies trying to reconstruct objects without 3D supervision [9,19,28].…”

“…The noteworthy aspect of the architecture is that it is a feed-forward network without any shortcut from lower layers, and it consists of several spatial downsamplings and channel upsamplings at the same time. This sort of architecture allows a multi-scale representation of the original image and has been shown to work better than the modern designs with skip connections when it comes to shape or texture representation [20,30].…”

“…Following [30], we extract a feature vector for each individual point by projecting the points onto the feature maps as illustrated in Figure 2 (top branch). Concretely, given an initial point cloud, we compute the 2D pixel coordinate of each point using camera intrinsics.…”

Multitask learning on monocular water images: Surface reconstruction and image synthesis

“…Another possible solution is to use the shape‐from‐shading (SFS) technique to recover water surfaces from videos, but this does not handle highlights and occlusions well and brings geometric distortions. Machine learning–based methods have shown promising progress in three‐dimensional (3D) reconstruction and depth estimation recently. Moreover, some machine learning–based fluid simulators have also been developed .…”