Deep Volumetric Video From Very Sparse Multi-view Performance Capture

Huang, Zeng; Li, Tianye; Chen, Weikai; Zhao, Yajie; Xing, Junliang; LeGendre, Chloe; Luo, Linjie; Ma, Chongyang; Li, Hao

doi:10.1007/978-3-030-01270-0_21

Cited by 116 publications

(95 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, due to possible inconsistency between the synthesized silhouettes, the subtraction operation of visual hull tends to excessively erode the reconstructed mesh. To further improve the output quality, we adopt a deep visual hull algorithm similar to Huang et al [20] with a greedy view sampling strategy so that the reconstruction results account for domain-specific clothed human body priors (Sec. 3.2).…”

Section: Methodsmentioning

confidence: 99%

“…In particular, we use a network structure based on [20]. At a high level, Huang et al [20] propose to map 2D images to a 3D volumetric field through a multi-view convolutional neural network. The 3D field encodes the probabilistic distribution of 3D points on the captured surface.…”

Section: Deep Visual Hull Predictionmentioning

confidence: 99%

“…The selected input silhouettes generated by our greedy view sampling algorithm are then fed into a deep visual hull network. The choice of our network design is similar to that of [20]. The main difference lies in the format of inputs.…”

Section: Deep Visual Hull Predictionmentioning

confidence: 99%

See 2 more Smart Citations

SiCloPe: Silhouette-Based Clothed People

Natsume

Saito

Huang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

203

153

View full text Add to dashboard Cite

We introduce a new silhouette-based representation for modeling clothed human bodies using deep generative models. Our method can reconstruct a complete and textured 3D model of a person wearing clothes from a single input picture. Inspired by the visual hull algorithm, our implicit representation uses 2D silhouettes and 3D joints of a body pose to describe the immense shape complexity and variations of clothed people. Given a segmented 2D silhouette of a person and its inferred 3D joints from the input picture, we first synthesize consistent silhouettes from novel view points around the subject. The synthesized silhouettes which are the most consistent with the input segmentation are fed into a deep visual hull algorithm for robust 3D shape prediction. We then infer the texture of the subject's back view using the frontal image and segmentation mask as input to a conditional generative adversarial network. Our experiments demonstrate that our silhouette-based model is an effective representation and the appearance of the back view can be predicted reliably using an image-to-image translation network. While classic methods based on parametric models often fail for single-view images of subjects with challenging clothing, our approach can still produce successful results, which are comparable to those obtained from multi-view input.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Deep Visual Hull Predictionmentioning

confidence: 99%

See 1 more Smart Citation

SiCloPe: Silhouette-Based Clothed People

Natsume

Saito

Huang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

203

153

View full text Add to dashboard Cite

show abstract

“…In order to replicate the success of 2D convolutional neural network to 3D domain, various forms of 3D representations have been explored. As a natural extension of 2D pixels, volumetric representation has been widely used in recent works on 3D reconstruction [29,30,34,6,13,31,36,10,33] due to its simplicity of implementation and compatibility with convolutional neural network. However, deep voxel generators are constrained by its resolution due to the data sparsity and computation cost of 3D convolution.…”

Section: Related Workmentioning

confidence: 99%

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Pan

Han

Chen

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

194

186

View full text Add to dashboard Cite

Reconstructing the 3D mesh of a general object from a single image is now possible thanks to the latest advances of deep learning technologies. However, due to the nontrivial difficulty of generating a feasible mesh structure, the stateof-the-art approaches [16,32] often simplify the problem by learning the displacements of a template mesh that deforms it to the target surface. Though reconstructing a 3D shape with complex topology can be achieved by deforming multiple mesh patches, it remains difficult to stitch the results to ensure a high meshing quality. In this paper, we present an end-to-end single-view mesh reconstruction framework that is able to generate high-quality meshes with complex topologies from a single genus-0 template mesh. The key to our approach is a novel progressive shaping framework that alternates between mesh deformation and topology modification. While a deformation network predicts the per-vertex translations that reduce the gap between the reconstructed mesh and the ground truth, a novel topology modification network is employed to prune the error-prone faces, enabling the evolution of topology. By iterating over the two procedures, one can progressively modify the mesh topology while achieving higher reconstruction accuracy. Moreover, a boundary refinement network is designed to refine the boundary conditions to further improve the visual quality of the reconstructed mesh. Extensive experiments demonstrate that our approach outperforms the current state-of-the-art methods both qualitatively and quantitatively, especially for the shapes with complex topologies.

show abstract

“…A large body of work exists to extract human representations from multiple input views or sensors, of which some recently use deep learning to extract 3D human representations [8,13,21]. While they intrinsically aren't designed to deal with monocular input as proposed, multi-view methods usually yield more complete and higher precision results as soon as several viewpoints are available, a useful feature we leverage for creating the 3D HUMANS dataset.…”

Section: Related Workmentioning

confidence: 99%

Moulding Humans: Non-Parametric 3D Human Shape Estimation From Single Images

Gabeur

Franco

Martín

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

173

111

View full text Add to dashboard Cite

In this paper, we tackle the problem of 3D human shape estimation from single RGB images. While the recent progress in convolutional neural networks has allowed impressive results for 3D human pose estimation, estimating the full 3D shape of a person is still an open issue. Model-based approaches can output precise meshes of naked under-cloth human bodies but fail to estimate details and un-modelled elements such as hair or clothing. On the other hand, non-parametric volumetric approaches can potentially estimate complete shapes but, in practice, they are limited by the resolution of the output grid and cannot produce detailed estimates. In this work, we propose a non-parametric approach that employs a double depth map to represent the 3D shape of a person: a visible depth map and a "hidden" depth map are estimated and combined, to reconstruct the human 3D shape as done with a "mould". This representation through 2D depth maps allows a higher resolution output with a much lower dimension than voxel-based volumetric representations. Additionally, our fully derivable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the 3D output. We train and quantitatively validate our approach on SURREAL and on 3D-HUMANS, a new photorealistic dataset made of semi-synthetic in-house images annotated with 3D ground truth surfaces.

show abstract

Deep Volumetric Video From Very Sparse Multi-view Performance Capture

Cited by 116 publications

References 72 publications

SiCloPe: Silhouette-Based Clothed People

SiCloPe: Silhouette-Based Clothed People

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Moulding Humans: Non-Parametric 3D Human Shape Estimation From Single Images

Contact Info

Product

Resources

About