Learning to Reconstruct People in Clothing From a Single RGB Camera

Abstract

Figure 1: We present a deep learning based approach to estimate personalized body shape, including hair and clothing, using a single RGB camera. The shapes shown above have been calculated using only 8 input images, and re-posed using SMPL. AbstractWe present Octopus, a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving with a reconstruction accuracy of 4 to 5mm, while being orders of magnitude faster than previous methods. From semantic segmentation images, our Octopus model reconstructs a 3D shape, including the parameters of SMPL plus clothing and hair in 10 seconds or less. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into poseinvariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, Octopus can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 5mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach. Code is available at [2]. * Work partly conducted during an internship at the Real Virtual Humans group of Max Planck for Informatics.

FACSIMILE: Fast and Accurate Scans From an Image in Less Than a Second

“…Single-photo body estimation methods typically bottleneck through fixed intermediate representations, which while enabling piecewise modeling, ultimately limit the amount of achievable detail. Some methods bottleneck through segmented images [21,15,38,10,33], others through estimated keypoints positions [7,26], and some through both [44,34,14,1]. All such methods permit too much ambiguity to allow for dense surface reconstruction.…”

360-Degree Textures of People in Clothing from a Single Image

“…Additionally, in Figure 5 we show comparison of our single-view method to the monocular video method of [6]. While our results look comparable and visually pleasing, should be noted that the method in [6] produces higher resolution reconstructions. The advantage of our method is that it requires just a single view.…”

Multi-Garment Net: Learning to Dress 3D People From Images

“…MGN: Multi-Garment Net The input to the model is a set of semantically segmented images, I = {I 0 , I 1 , ..., I F − 1}, and corresponding 2D joint estimates, J = {J 0 , J 1 , ..., J F −1}, where F is the number of images used to make the prediction. Following [20,3], we abstract away the appearance information in RGB images and extract semantic garment segmentation [20] to reduce the risk of over-fitting, albeit at the cost of disregarding useful shading signal. For simplicity, let now θ denote both the joint angles θ and translation t.…”

“…• 2D segmentation loss: Unlike [3] we do not optimize silhouette overlap, instead we jointly optimize the projected per-garment segmentation against the input segmentation mask. This ensures that each garment explains its corresponding mask in the image:…”

“…We held out 70 scans for testing and use the rest for training. Similar to [3,5], we also restrict our setting to the scenario where the person is turning around in front of the camera. We register the scans using multi-mesh registration, SMPL+G.…”