HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling

Cai, Zhongang; Ren, Daxuan; Zeng, Ailing; Lin, Zhengyu; Yu, Tao; Wang, Wenjia; Fan, Xiangyu; Gao, Yang; Yu, Yifan; Pan, Liang; Hong, Fangzhou; Zhang, Mingyuan; Loy, Chen Change; Yang, Lei; Liu, Ziwei

doi:10.1007/978-3-031-20071-7_33

Cited by 44 publications

(11 citation statements)

References 107 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is due to a large amount of data necessary to sufficiently sample the model space. However, recent progress in dataset [HYH*20, CRZ*22] acquisition may now enable the building of such a model. Another unsolved fundamental problem is the tracking of topological changes ( e.g .…”

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

Tretschk

Navami

Mallikarjun

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

3D reconstruction of deformable (or non‐rigid) scenes from a set of monocular 2D image observations is a long‐standing and actively researched area of computer vision and graphics. It is an ill‐posed inverse problem, since—without additional prior assumptions—it permits infinitely many solutions leading to accurate projection to the input 2D images. Non‐rigid reconstruction is a foundational building block for downstream applications like robotics, AR/VR, or visual content creation. The key advantage of using monocular cameras is their omnipresence and availability to the end users as well as their ease of use compared to more sophisticated camera set‐ups such as stereo or multi‐view systems. This survey focuses on state‐of‐the‐art methods for dense non‐rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views. It reviews the fundamentals of 3D reconstruction and deformation modeling from 2D image observations. We then start from general methods—that handle arbitrary scenes and make only a few prior assumptions—and proceed towards techniques making stronger assumptions about the observed objects and types of deformations (e.g. human faces, bodies, hands, and animals). A significant part of this STAR is also devoted to classification and a high‐level comparison of the methods, as well as an overview of the datasets for training and evaluation of the discussed techniques. We conclude by discussing open challenges in the field and the social aspects associated with the usage of the reviewed methods.

show abstract

Section: State‐of‐the‐art Methodsmentioning

confidence: 99%

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

Tretschk

Navami

Mallikarjun

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…In fact, using the skill label as a condition brings a lot of flexibility, which makes our method compatible with various input sources. For instance, methods that directly regress the skeleton rotation [Shi [Cai et al 2022;Dou et al 2022;Li et al 2023Li et al , 2021bWan et al 2021;] of the parametric model, e.g., SMPL [Loper et al 2015], from the video, can be directly applied to our labeling. Moreover, skill labels are typically aligned with the language of skill descriptions, which could open the door for language-guide motion control.…”

Section: A2 Skill Labelmentioning

confidence: 99%

C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters

Dou,

Chen,

Fan

et al. 2023

SIGGRAPH Asia 2023 Conference Papers

View full text Add to dashboard Cite

Figure 1: Our framework enables physically simulated characters to master highly varied and extensive skills with high efficiency and effectiveness. Notably, it offers an explicit control handle for directly specifying the desired skill from a diverse and extensive set of skills. Here, a character is instructed by the user to perform a sequence of skills, including kick, jump attack, sword bash, shield bash, and finally, roaring.

show abstract

“…Currently there exist many datasets providing RGB images and 3D pose annotations in both singleperson and multi-person scenarios. Given the ease of data collection, there are numerous single-person 2D [Joo et al 2021;Kolotouros et al 2019;Lin et al 2014] and 3D datasets [Cai et al 2022;Chatzitofis et al 2020;Fieraru et al 2021a,b;Ionescu et al 2014;Mehta et al 2017;Ofli et al 2013;Sigal et al 2010;Trumble et al 2017;Von Marcard et al 2018;Yoon et al 2021], featuring diverse actors, actions, and modalities. However, training and validation data are relatively scarce in multi-person scenarios.…”

Section: Related Workmentioning

confidence: 99%

Reconstructing Close Human Interactions from Multiple Views

Shuai,

Yu,

Zhou

et al. 2023

ACM Trans. Graph.

View full text Add to dashboard Cite

This paper addresses the challenging task of reconstructing the poses of multiple individuals engaged in close interactions, captured by multiple calibrated cameras. The difficulty arises from the noisy or false 2D keypoint detections due to inter-person occlusion, the heavy ambiguity in associating keypoints to individuals due to the close interactions, and the scarcity of training data as collecting and annotating motion data in crowded scenes is resource-intensive. We introduce a novel system to address these challenges. Our system integrates a learning-based pose estimation component and its corresponding training and inference strategies. The pose estimation component takes multi-view 2D keypoint heatmaps as input and reconstructs the pose of each individual using a 3D conditional volumetric network. As the network doesn't need images as input, we can leverage known camera parameters from test scenes and a large quantity of existing motion capture data to synthesize massive training data that mimics the real data distribution in test scenes. Extensive experiments demonstrate that our approach significantly surpasses previous approaches in terms of pose accuracy and is generalizable across various camera setups and population sizes. The code is available on our project page: https://github.com/zju3dv/CloseMoCap.

show abstract

HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling

Cited by 44 publications

References 107 publications

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characters

Reconstructing Close Human Interactions from Multiple Views

Contact Info

Product

Resources

About