DeepGarment : 3D Garment Shape Estimation from a Single Image

Abstract

These authors contributed equallyFigure 1: Garment 3D shape estimation using our CNN model and a single-view. From left to right: real-life images capturing a person wearing a T-shirt, segmented and cut-out garments and 3D estimations of the shape.Abstract 3D garment capture is an important component for various applications such as free-view point video, virtual avatars, online shopping, and virtual cloth fitting. Due to the complexity of the deformations, capturing 3D garment shapes requires controlled and specialized setups. A viable alternative is image-based garment capture. Capturing 3D garment shapes from a single image, however, is a challenging problem and the current solutions come with assumptions on the lighting, camera calibration, complexity of human or mannequin poses considered, and more importantly a stable physical state for the garment and the underlying human body. In addition, most of the works require manual interaction and exhibit high run-times. We propose a new technique that overcomes these limitations, making garment shape estimation from an image a practical approach for dynamic garment capture. Starting from synthetic garment shape data generated through physically based simulations from various human bodies in complex poses obtained through Mocap sequences, and rendered under varying camera positions and lighting conditions, our novel method learns a mapping from rendered garment images to the underlying 3D garment model. This is achieved by training Convolutional Neural Networks (CNN-s) to estimate 3D vertex displacements from a template mesh with a specialized loss function. We illustrate that this technique is able to recover the global shape of dynamic 3D garments from a single image under varying factors such as challenging human poses, self occlusions, various camera poses and lighting conditions, at interactive rates. Improvement is shown if more than one view is integrated. Additionally, we show applications of our method to videos.

Video Based Reconstruction of 3D People Models

“…Only very few works predict human shape from images using learning methods since images annotated with ground truth shape, pose and clothing geometry are hardly available. A few exceptions are the approach of [20] that predicts shape from silhouettes using a neural network and [18] that predicts garment geometry from a single image. Predictions in [20] are restricted to model shape space and tend to look over-smooth; only garments seen in the dataset can be recovered in [18].…”

“…A few exceptions are the approach of [20] that predicts shape from silhouettes using a neural network and [18] that predicts garment geometry from a single image. Predictions in [20] are restricted to model shape space and tend to look over-smooth; only garments seen in the dataset can be recovered in [18]. Recent works leverage 2D annotations to train networks for the task of 3D pose estimation [42,53,84,65,68,56].…”

DeepWrinkles: Accurate and Realistic Clothing Modeling

“…[46] learns bags of dynamical systems to represent and recognize repeating patterns in wrinkle deformations. In DeepGarment [13] the global shape and low frequency details are reconstructed from a single segmented image using a CNN but no retargeting is possible.…”

GarNet: A Two-Stream Network for Fast and Accurate 3D Cloth Draping

“…3D human body/cloth reconstruction. 3D body shapes/cloth are modeled from RGB/RGBD cameras in [49,43,42,15,2,1,47,48] while garment and surface reconstruction methods from images are addressed in surface/wrinkle reconstruction from images [9,3,35]. Moreover, generative models reconstruct cloths in [25,16].…”