Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Huang, Buzhen; Pan, Liang; Yang, Yadong; Ju, Jingyi; Wang, Yangang

doi:10.1109/cvpr52688.2022.00631

Cited by 23 publications

(5 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the advent of multi-modal models, Mo-tionCLIP (Tevet et al 2022) harnessed the latent space of CLIP (Huang et al 2019). The diffusion model emerged as a novel tool, with MotionDiffuse (Zhang et al 2022) and ReMoDiffuse (Zhang et al 2023b) generating vivid, semantically consistent, and high-fidelity motion sequences. This exciting domain continues to evolve, pushing the boundaries of text-driven motion synthesis.…”

Section: Related Work Text To Motion Methodsmentioning

confidence: 99%

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

Fan,

Ji,

et al. 2024

AAAI

View full text Add to dashboard Cite

In the dynamic field of film and game development, the emergence of human motion synthesis methods has revolutionized avatar animation. Traditional methodologies, typically reliant on single modality inputs like text or audio, employ modality-specific model frameworks, posing challenges for unified model deployment and application. To address this, we propose Everything2Motion, a unified model framework. Everything2Motion consists of three key modules. The Input-Output Modality Modulation module tailors structures for specific multimodal inputs, eliminating the need for modality-specific frameworks. The Query-aware Autoencoder, based on the transformer encoder-decoder architecture, enables efficient latent motion generation. Lastly, the Prior Motion Distillation Decoder, a pretrained module, enhances the final skeleton sequence's naturalness and fluidity. Comprehensive experiments on several public datasets demonstrate the effectiveness of Everything2Motion, highlighting its potential for practical applications and setting a new benchmark in human motion synthesis.

show abstract

Section: Related Work Text To Motion Methodsmentioning

confidence: 99%

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

Fan,

Ji,

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…While multi‐view camera algorithms [AARS13, BSC13, DFJ*22] have achieved higher accuracy, they often require laborious camera system calibration. Mono‐camera approaches with optimization techniques [BKL*16, KPD19] and neural networks [PZDD17,WLLL22,HPY*22] lack depth information and struggle to track global translations. Despite offering an additional depth channel, RGBD‐based solutions [BMB*11, MSS*17, YZ21] are hindered by limited camera resolution and a field of view (FOV), which makes them impractical for product‐level applications.…”

Section: Related Workmentioning

confidence: 99%

MOVIN: Real‐time Motion Capture using a Single LiDAR

Jang,

Yang,

Jang

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

Recent advancements in technology have brought forth new forms of interactive applications, such as the social metaverse, where end users interact with each other through their virtual avatars. In such applications, precise full‐body tracking is essential for an immersive experience and a sense of embodiment with the virtual avatar. However, current motion capture systems are not easily accessible to end users due to their high cost, the requirement for special skills to operate them, or the discomfort associated with wearable devices. In this paper, we present MOVIN, the data‐driven generative method for real‐time motion capture with global tracking, using a single LiDAR sensor. Our autoregressive conditional variational autoencoder (CVAE) model learns the distribution of pose variations conditioned on the given 3D point cloud from LiDAR. As a central factor for high‐accuracy motion capture, we propose a novel feature encoder to learn the correlation between the historical 3D point cloud data and global, local pose features, resulting in effective learning of the pose prior. Global pose features include root translation, rotation, and foot contacts, while local features comprise joint positions and rotations. Subsequently, a pose generator takes into account the sampled latent variable along with the features from the previous frame to generate a plausible current pose. Our framework accurately predicts the performer's 3D global information and local joint details while effectively considering temporally coherent movements across frames. We demonstrate the effectiveness of our architecture through quantitative and qualitative evaluations, comparing it against state‐of‐the‐art methods. Additionally, we implement a real‐time application to showcase our method in real‐world scenarios. MOVIN dataset is available at https://movin3d.github.io/movin_pg2023/https://movin3d.github.io/movin_pg2023/">https://movin3d.github.io/movin_pg2023/.

show abstract

“…It means that the possible set of actions that an actor can perform should concern surrounding environment Tahtali 2018, 2021). Affordance can be considered in various tasks that involve the visual understanding of the human, such as hand pose estimation (Grady et al 2021;Corona et al 2020;Williams and Mahapatra 2019), 3D human avatar generation (Li et al 2019b;Zhang et al 2020a;Hassan et al 2021), 3D pose generation (Wang et al 2019), motion prediction (Huang et al 2022;Cao et al 2020;Wang et al 2021), object affordance prediction (Do, Nguyen, and Reid 2018;Kim and Sukhatme 2014;Fang et al 2018), and shape estimation (Clever et al 2020).…”

Section: Related Workmentioning

confidence: 99%

Pose-Guided 3D Human Generation in Indoor Scene

Kim

Kang

Park

et al. 2023

AAAI

View full text Add to dashboard Cite

In this work, we address the problem of scene-aware 3D human avatar generation based on human-scene interactions. In particular, we pay attention to the fact that physical contact between a 3D human and a scene (i.e., physical human-scene interactions) requires a geometrical alignment to generate natural 3D human avatar. Motivated by this fact, we present a new 3D human generation framework that considers geometric alignment on potential contact areas between 3D human avatars and their surroundings. In addition, we introduce a compact yet effective human pose classifier that classifies the human pose and provides potential contact areas of the 3D human avatar. It allows us to adaptively use geometric alignment loss according to the classified human pose. Compared to state-of-the-art method, our method can generate physically and semantically plausible 3D humans that interact naturally with 3D scenes without additional post-processing. In our evaluations, we achieve the improvements with more plausible interactions and more variety of poses than prior research in qualitative and quantitative analysis. Project page: https://bupyeonghealer.github.io/phin/.

show abstract

Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture

Cited by 23 publications

References 49 publications

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

Everything2Motion: Synchronizing Diverse Inputs via a Unified Framework for Human Motion Synthesis

MOVIN: Real‐time Motion Capture using a Single LiDAR

Pose-Guided 3D Human Generation in Indoor Scene

Contact Info

Product

Resources

About