Leveraging depth cameras and wearable pressure sensors for full-body kinematics and dynamics capture

Abstract

Figure 1: Our system automatically and accurately reconstructs full-body kinematics and dynamics data using input data captured by three depth cameras and a pair of pressure-sensing shoes. (top) reference image data; (bottom) the reconstructed full-body poses and contact forces (red arrows) and torsional torques (yellow arrows) applied at the center of pressure. AbstractWe present a new method for full-body motion capture that uses input data captured by three depth cameras and a pair of pressuresensing shoes. Our system is appealing because it is low-cost, non-intrusive and fully automatic, and can accurately reconstruct both full-body kinematics and dynamics data. We first introduce a novel tracking process that automatically reconstructs 3D skeletal poses using input data captured by three Kinect cameras and wearable pressure sensors. We formulate the problem in an optimization framework and incrementally update 3D skeletal poses with observed depth data and pressure data via iterative linear solvers. The system is highly accurate because we integrate depth data from multiple depth cameras, foot pressure data, detailed full-body geometry, and environmental contact constraints into a unified framework. In addition, we develop an efficient physics-based motion reconstruction algorithm for solving internal joint torques and contact forces in the quadratic programming framework. During reconstruction, we leverage Newtonian physics, friction cone constraints, contact pressure information, and 3D kinematic poses obtained from the kinematic tracking process to reconstruct full-body dynamics data. We demonstrate the power of our approach by capturing a wide range of human movements and achieve state-of-theart accuracy in our comparison against alternative systems.

Real-Time Human Motion Capture with Multiple Depth Cameras

“…They report a median joint prediction error of approximately 15 cm on a T-pose sequence. Zhang et al [9] combine the depth data with wearable pressure sensors to estimate shape and track human subjects at 6 fps. In this work we demonstrate how the traditional building blocks can be effectively merged to achieve the state-of-the-art performance in multiview depthbased pose estimation.…”

Robust Articulated‐ICP for Real‐Time Hand Tracking

“…Note that there is a widespread belief [WZC12, ZSZ*14, QSW*14] that ICP‐like techniques are too local and prone to local minima to successfully deal with fast articulated motion. One of our contributions is to show this commonly held belief should be re‐considered.…”

Dance motion capture and composition using multiple RGB and depth sensors

“…Given a labeled training set of image patches, Plagemann et al 16 used local shape descriptors to detect the salient body parts only. Shotton et al Backed by an existing database, the hybrid approaches [19][20][21][22][23] try to improve the tracking accuracy by complementing the generative methods (i.e. the optimization problems) with the discriminative methods (i.e.…”