From Image to Stability: Learning Dynamics from Human Pose

Abstract

We propose and validate two end-to-end deep learning architectures to learn foot pressure distribution maps (dynamics) from 2D or 3D human pose (kinematics). The networks are trained using 1.36 million synchronized pose+pressure data pairs from 10 subjects performing multiple takes of a 5-minute long choreographed Taiji sequence. Using leave-one-subject-out cross validation, we demonstrate reliable and repeatable foot pressure prediction, setting the first baseline for solving a non-obvious pose to pressure cross-modality mapping problem in computer vision. Furthermore, we compute and quantitatively validate Center of Pressure (CoP) and Base of Support (BoS), two key components for stability analysis, from the predicted foot pressure distributions.

Image-based Stability Quantification

“…In this work, ground truth data for training and evaluation is provided by simultaneously recorded video, motion capture, and foot pressure sensor data (Figure 2) from the PSU-TMM100 dataset [10]. This is the only available dataset that includes synchronized, sensor-measured recordings of these three modalities, making it a unique and valuable resource for learning to predict stability from imagery.…”

“…For image-based pose, we use the open-source OpenPose (OP) network, which provides 25 joint estimates (Figure 3A). There are 12 joints in common between GT (Figure 3B) and OP, and we apply the BioPose correction network from [10], [49] to the OP estimates of these 12 joints to derive a set of BioPose (BP) joints (Figure 3C). Finally, those 12 BP joints are combined with the remaining 13 OP joints and referred to as HybridPose (HP) (Figure 3D).…”

“…2) CoP and BoS Prediction: We use the PressNet-Simple 3D (PNS3) network from [10] for image-based foot pressure predictions. While OpenPose joint data are shown in [10] to be the best input for predicting foot pressure, we evaluate motion capture, HybridPose, and OpenPose data for foot localization for all takes of PSU-TMM100. The calculation of CoP and BoS follows the same calculation steps as the ground truth process but replacing inputs with image-based data.…”

“…Yet, little work has been done in image-based mapping of body kinematics (pose) to dynamics (foot pressure/force) output. In an initial study [10], we demonstrated the feasibility of predicting foot pressure from images of human pose. We take a step further in this work to explore the feasibility of predicting BoS, CoP, and CoM measures exclusively from visual input, and using these image-based estimates to compute two classic human stability metrics: CoM to BoS, and CoM to CoP (Figure 1, Table I).…”

“…The main contributions of this work include: 1) a thorough and complete quantitative comparison study assessing two stability metrics (Table I) computed using component values CoP, BoS, and CoM obtained from either imagebased or sensor-based (ground truth) measurements (2 3 = 8 combinations evaluated); 2) developing and validating an image-based machine learning algorithm for CoM estimation from image data ; 3) finding that a fully image-based approach (eliminating the need for foot pressure sensors and motion capture) pro-Fig. 1: Stability components and two stability metrics (CoM-toBoS and CoMtoCoP) relative to localized foot pressure with CoP (red star), CoM (blue star), BoS (green border), foot pressure (yellow/red/brown gradations), and metrics (cyan lines) [10].…”