Capturing and Inferring Dense Full-Body Human-Scene Contact

Huang, Chun-Hao P.; Yi, Hongwei; Hoschle, Markus; Safroshkin, Matvey; Alexiadis, Tsvetelina; Polikovsky, Senya; Scharstein, Daniel; Black, Michael J.

doi:10.1109/cvpr52688.2022.01292

Cited by 77 publications

(29 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent approaches begin to tackle modeling and synthesizing human interactions within 3D scenes, or with objects. Most of the researches focus on statically posing humans within the given 3D environment [16,24,69,71], by generating human scene interaction poses from various types of input including object semantics [17], images [21,23,64,65,68], and text descriptions [49,72].…”

Section: Related Workmentioning

confidence: 99%

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

Lee¹,

Joo²

2023

Preprint

View full text Add to dashboard Cite

show abstract

Section: Related Workmentioning

confidence: 99%

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

Lee¹,

Joo²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Most related to our work are two recent datasets. The RICH dataset [21], fits SMPL-X bodies to multi-view RGB videos taken both indoors and outdoors. The method uses a detailed 3D scan of the scene and models the contact between the body and the world.…”

Section: Related Workmentioning

confidence: 99%

“…This makes these datasets ill-suited as evaluation benchmarks. The recent RICH dataset [21] addresses many of these issues with indoor and outdoor scenes, accurate multi-view capture of SMPL-X, 3D scene scans, and human-scene contact. It is not appropriate for our task, however, as it does not include object manipulation.…”

Section: Joint Modeling Of Humans and Scenesmentioning

confidence: 99%

“…Appear. Objects Motion Image Hands HumanEva [52] 56 Human3.6M [23] 165 AMASS [35] 11265 GRAB [54] 1334 3DPW [36] 60 GTA-IM [5] 119 SAIL-VOS [20] 201 PiGraphs [51] 63 PROX [15] 20 RICH [21] 142 BEHAVE [3] 321 InterCap (ours) 223 An alternative approach is the one of GTA-IM [5] and SAIL-VOS [20], which generate human-scene interaction data using either 3D graphics or 2D videos. They feature highaccuracy ground truth but lack visual realism.…”

Section: Namementioning

confidence: 99%

See 1 more Smart Citation

InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction

Huang

Taheri

Black

et al. 2022

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Humans constantly interact with daily objects to accomplish tasks. To understand such interactions, computers need to reconstruct these from cameras observing whole-body interaction with scenes. This is challenging due to occlusion between the body and objects, motion blur, depth/scale ambiguities, and the low image resolution of hands and graspable object parts. To make the problem tractable, the community focuses either on interacting hands, ignoring the body, or on interacting bodies, ignoring hands. The GRAB dataset addresses dexterous whole-body interaction but uses marker-based MoCap and lacks images, while BEHAVE captures video of body-object interaction but lacks hand detail. We address the limitations of prior work with InterCap, a novel method that reconstructs interacting whole-bodies and objects from multi-view RGB-D data, using the parametric whole-body model SMPL-X and known object meshes. To tackle the above challenges, InterCap uses two key observations: (i) Contact between the hand and object can be used to improve the pose estimation of both. (ii) Azure Kinect sensors allow us to set up a simple multi-view RGB-D capture system that minimizes the effect of occlusion while providing reasonable inter-camera synchronization. With this method we capture the InterCap dataset, which contains 10 subjects (5 males and 5 females) interacting with 10 objects of various sizes and affordances, including contact with the hands or feet. In total, InterCap has 223 RGB-D videos, resulting in 67,357 multi-view frames, each containing 6 RGB-D images. Our method provides pseudo ground-truth body meshes and objects for each video frame. Our InterCap method and dataset fill an important gap in the literature and support many research directions. Our data and code are available for research purposes at https://intercap.is.tue.mpg.de.

show abstract

“…However, the contact sequence is planned or annotated usually manually based on experiences. On the other hand, especially in the computer vision and graphics field, many datasets incorporating contacts are available, for example, BEHAVE ( Bhatnagar et al, 2022 ), RICH ( Huang et al, 2022 ), HuMoD ( Wojtusch and von Stryk, 2015 ), and MMM ( Mandery et al, 2016 ). While they are very useful to broaden the range of motion creation for animation or digital human, data with contact forces need to be reinforced, and their physical plausibility is to be improved for robotic motion generation yet.…”

Section: Challenges For Contact-rich Motions Through Icphmentioning

confidence: 99%

Towards understanding and synthesis of contact-rich anthropomorphic motions through interactive cyber-physical human

Yoshida

2022

Front. Robot. AI

View full text Add to dashboard Cite

This article presents perspective on the research challenge of understanding and synthesizing anthropomorphic whole-body contact motions through a platform called “interactive cyber-physical human (iCPH)” for data collection and augmentation. The iCPH platform combines humanoid robots as “physical twins” of human and “digital twins” that simulates humans and robots in cyber-space. Several critical research topics are introduced to address this challenge by leveraging the advanced model-based analysis together with data-driven learning to exploit collected data from the integrated platform of iCPH. Definition of general description is identified as the first topic as a common basis of contact motions compatible to both humans and humanoids. Then, we set continual learning of a feasible contact motion network as the second challenge by benefiting from model-based approach and machine learning bridged by the efficient analytical gradient computation developed by the author and his collaborators. The final target is to establish a high-level symbolic system allowing automatic understanding and generation of contact motions in unexperienced environments. The proposed approaches are still under investigation, and the author expects that this article triggers discussions and further collaborations from different research communities, including robotics, artificial intelligence, neuroscience, and biomechanics.

show abstract

Capturing and Inferring Dense Full-Body Human-Scene Contact

Cited by 77 publications

References 65 publications

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments

InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interaction

Towards understanding and synthesis of contact-rich anthropomorphic motions through interactive cyber-physical human

Contact Info

Product

Resources

About