Three-Dimensional Reconstruction of Human Interactions

Fieraru, Mihai; Zanfir, Mihai; Oneata, Elisabeta; Popa, Alin-Ionut; Olaru, Vlad; Sminchisescu, Cristian

doi:10.1109/cvpr42600.2020.00724

Cited by 76 publications

(36 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Datasets for 3D human motion and interactions. A large number of datasets focus on 3D human pose and motion from third-person views [16,23,27,30,31,43,52,57,71,74,75,84,90]. For example, Human3.6M [27] and AMASS [51] use optical marker-based motion capture to collect large amounts of high-quality 3D motion sequences; they are limited to constrained studio setups and images -when available -are polluted by marker data.…”

Section: Related Workmentioning

confidence: 99%

“…The Panoptic Studio dataset [30][31][32]79] reconstructs interactions between people using a multi-view camera system; it provides annotations for body and hand 3D joints plus facial landmarks. CHI3D [16] focuses on close human-human contacts, using a motion capture system to extract ground-truth 3D skeletons. 3DPW [74] reconstructs both the 3D shape and motion of people "in-the-wild" by fitting SMPL [48] to IMU data and RGB images captured with a hand-held camera, without reconstructing the 3D environment.…”

Section: Related Workmentioning

confidence: 99%

“…A reason for this limited attention is the lack of data. Most human motion datasets do not provide egocentric frames [16,23,27,30,31,52,71,90], and most capture only one subject at a time [23,27]. On the other hand, existing egocentric human motion datasets have limitations in terms of annotations, scale and diversity [22,47,55,69,70,80,85].…”

mentioning

confidence: 99%

See 2 more Smart Citations

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Zhang¹,

Qian²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Zhang¹,

Qian²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

“…There is a separate research direction on domainspecific mesh deformation representation, e.g., SMPL/S-TAR human models (Loper et al 2015;Osman, Bolkart, and Black 2020), wrinkle-enhanced cloth meshes (Lahner, Cremers, and Tung 2018), and skeletal skinning meshes (Xu et al 2020). There are even prior works (Fieraru et al 2020(Fieraru et al , 2021Muller et al 2021) on collision detection and handling for human bodies. These domain-specific methods are typically more accurate than our representation, but by assuming general meshes, our representation can be applied to multiple domains as shown in Sec.…”

Section: Related Workmentioning

confidence: 99%

Active Learning of Neural Collision Handler for Complex 3D Mesh Deformations

Tan,

Pan,

Smith

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a robust learning algorithm to detect and handle collisions in 3D deforming meshes. Our collision detector is represented as a bilevel deep autoencoder with an attention mechanism that identifies colliding mesh sub-parts. We use a numerical optimization algorithm to resolve penetrations guided by the network. Our learned collision handler can resolve collisions for unseen, high-dimensional meshes with thousands of vertices. To obtain stable network performance in such large and unseen spaces, we progressively insert new collision data based on the errors in network inferences. We automatically label these data using an analytical collision detector and progressively fine-tune our detection networks. We evaluate our method for collision handling of complex, 3D meshes coming from several datasets with different shapes and topologies, including datasets corresponding to dressed and undressed human poses, cloth simulations, and human hand poses acquired using multiview capture systems. Our approach outperforms supervised learning methods and achieves 93.8 − 98.1% accuracy compared to the groundtruth by analytic methods. Compared to prior learning methods, our approach results in a 5.16% − 25.50% lower false negative rate in terms of collision checking and a 9.65% − 58.91% higher success rate in collision handling.

show abstract

“…Véges et al [38] make use of a monocular depth prediction network pretrained on various indoor and outdoor datasets to help with absolute person distance estimation. Finally, some recent works also consider the depth relations among people: Jiang et al [39] optimize the depth ordering by occlusion cues, while Fieraru et al [40] explicitly localize contact points between people to help with coherent reconstruction. In contrast, we perform our estimation for each person independently.…”

Section: Scale and Distance Estimationmentioning

confidence: 99%

MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation

Sárándi,

Linder,

Arras

et al. 2020

Preprint

View full text Add to dashboard Cite

Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust (MeTRo) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D. Our code is publicly available.

show abstract

Three-Dimensional Reconstruction of Human Interactions

Cited by 76 publications

References 34 publications

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices

Active Learning of Neural Collision Handler for Complex 3D Mesh Deformations

MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation

Contact Info

Product

Resources

About