BodySLAM: Joint Camera Localisation, Mapping, and Human Motion Tracking

Henning, Dorian F.; Laidlow, Tristan; Leutenegger, Stefan

doi:10.1007/978-3-031-19842-7_38

Cited by 12 publications

(2 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several current methods focus on positioning humans in a pre‐scanned 3D scene [GMSP21; HYH*22; HCTB19] and on simultaneous estimation of human poses and objects humans interact with [WY21; CHY*19; YHT*22]. Different setups from ours assume an RGB‐D sensor [ZZB*21b] or a moving camera [ZMZ*22; LYZ*21; HLL22; LBX*22] to estimating the scene geometry. Recent methods integrate physics‐based constraints into monocular 3D human motion capture and mitigate foot‐floor penetration and other severe artefacts [SGXT20; SGX*21].…”

Section: Related Workmentioning

confidence: 99%

Scene‐Aware 3D Multi‐Human Motion Capture from a Single Camera

Luvizon

Habermann

Golyanik

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

In this work, we consider the problem of estimating the 3D position of multiple humans in a scene as well as their body shape and articulation from a single RGB video recorded with a static camera. In contrast to expensive marker‐based or multi‐view systems, our lightweight setup is ideal for private users as it enables an affordable 3D motion capture that is easy to install and does not require expert knowledge. To deal with this challenging setting, we leverage recent advances in computer vision using large‐scale pre‐trained models for a variety of modalities, including 2D body joints, joint angles, normalized disparity maps, and human segmentation masks. Thus, we introduce the first non‐linear optimization‐based approach that jointly solves for the 3D position of each human, their articulated pose, their individual shapes as well as the scale of the scene. In particular, we estimate the scene depth and person scale from normalized disparity predictions using the 2D body joints and joint angles. Given the per‐frame scene depth, we reconstruct a point‐cloud of the static scene in 3D space. Finally, given the per‐frame 3D estimates of the humans and scene point‐cloud, we perform a space‐time coherent optimization over the video to ensure temporal, spatial and physical plausibility. We evaluate our method on established multi‐person 3D human pose benchmarks where we consistently outperform previous methods and we qualitatively demonstrate that our method is robust to in‐the‐wild conditions including challenging scenes with people of different sizes. Code: https://github.com/dluvizon/scene‐aware‐3d‐multi‐human

show abstract

Section: Related Workmentioning

confidence: 99%

Scene‐Aware 3D Multi‐Human Motion Capture from a Single Camera

Luvizon

Habermann

Golyanik

et al. 2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…In contrast, our work recovers human trajectories for in-the-wild videos, in which camera motion is uncontrolled, and the scene reconstruction is limited or non-existent. [15] operate on monocular sequences, but the extent of results is limited to a single unoccluded person slowly walking in an indoor studio. We demonstrate our approach on PoseTrack, a complex in-the-wild dataset, which includes videos with a large number of people in various environments.…”

Section: Related Workmentioning

confidence: 99%

Decoupling Human and Camera Motion from Videos in the Wild

Ye¹,

Pavlakos²,

Malik³

et al. 2023

Preprint

View full text Add to dashboard Cite

Human Motion in the World Frame Figure 1. 4D Reconstruction of People from Videos in-the-Wild. We present SLAHMR: Simultaneous Localization And Human Mesh Recovery, a method that given a video of moving people (top), recovers the global trajectories of all people and cameras in the world coordinate space (bottom). We combine geometric insights, which determine relative camera motion, with learned human motion priors, which constrain a person's plausible displacement between frames, to position the people and cameras in the shared world frame through time. Our method can recover the global trajectories of all detected people from in-the-wild videos with uncontrolled camera and human motion. Please see the project page to see the full video results.

show abstract