Estimating 3D Motion and Forces of Person-Object Interactions From Monocular Video

Abstract

In this paper, we introduce a method to automatically reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person and the object, contact positions, and forces and torques actuated by the human limbs. The main contributions of this work are three-fold. First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions. This is cast as a large-scale trajectory optimization problem. Second, we develop a method to automatically recognize from the input video the position and timing of contacts between the person and the object or the ground, thereby significantly simplifying the complexity of the optimization. Third, we validate our approach on a recent MoCap dataset with ground truth contact forces and demonstrate its performance on a new dataset of Internet videos showing people manipulating a variety of tools in unconstrained environments.* Please see our project webpage [2] for trained models, data and code.

Resolving 3D Human Pose Ambiguities With 3D Scene Constraints

“…They then refine poses in a global optimization scheme over all frames incorporating contact and temporal constraints, as well as collision constraints, using a collision model comprised of shape primitives similar to [10,47]. More recently, [39] introduced a method to estimate contact positions, forces and torques actuated by the human limbs during human-object interaction.…”

Neural monocular 3D human motion capture with physical awareness

“…Zell et al [2017] estimate 3D human poses along with the inner and exterior forces from images for object lifting and walking. Li et al [2019] regress human and object poses in 3D along with forces and torques exerted by human limbs from a monocular video and an object prior. They focus on instruments with grips and recognise contacts between a person and an object (i.e., the instrument or the ground) to facilitate the trajectory-optimisation problem.…”

Physically Consistent Whole-Body Kinematics Assessment Based on an RGB-D Sensor. Application to Simple Rehabilitation Exercises

“…However, such approaches are not yet real-time or accurate enough to reliably estimate a patient's joint angles and thus are not accessible to the clinical community. Progress is expected in terms of accuracy thanks to new methods that combine machine learning and model-based approaches [9]. Devices based on RGB-depth sensors with embedded skeleton tracking algorithms are already available to the wider public.…”