Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Lee, Kyoungoh; Lee, Sang Hoon; Lee, Sanghoon

doi:10.1007/978-3-030-01234-2_8

Cited by 195 publications

(129 citation statements)

References 45 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Structure-aware network architectures have also been used in 3D pose estimation from images [16,29,21,17,31]. [17] and [31] both learn a structured latent space.…”

Section: Related Workmentioning

confidence: 99%

“…[21] exploit structure only implicitly by encoding the poses into distance matrices which then serve as inputs and outputs of the network. [16] and [29] are closest to our work as they explicitly modify the network to account for skeletal structure, either via the loss function [29], or using a sequence of LSTM cells for each joint in the skeleton [16]. [16] introduces many new layers into the architecture and needs hyper-parameter tuning to be most effective.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Structured Prediction Helps 3D Human Motion Modelling

Aksan

Kaufmann

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

197

151

View full text Add to dashboard Cite

Figure 1: We introduce a structured prediction layer (SPL) to the task of 3D human motion modelling. The SP-layer explicitly decomposes the pose into individual joints and can be interfaced with a variety of baseline architectures. We show that on H3.6M and a recent, much larger dataset, AMASS, a variety of baseline models benefit when augmented with an SP-layer. AbstractHuman motion prediction is a challenging and important task in many computer vision application domains. Existing work only implicitly models the spatial structure of the human skeleton. In this paper, we propose a novel approach that decomposes the prediction into individual joints by means of a structured prediction layer that explicitly models the joint dependencies. This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function. The proposed layer is agnostic to the underlying network and can be used with existing architectures for motion modelling. Prior work typically leverages the H3.6M dataset. We show that some state-of-the-art techniques do not perform well when trained and tested on AMASS, a recently released dataset 14 times the size of H3.6M. Our experiments indicate that the proposed layer increases the performance of motion forecasting irrespective of the base network, jointangle representation, and prediction horizon. We furthermore show that the layer also improves motion predictions qualitatively. We make code and models publicly available at https://ait.ethz.ch/projects/2019/spl.

show abstract

“…Structure-aware network architectures have also been used in 3D pose estimation from images [16,29,21,17,31]. [17] and [31] both learn a structured latent space.…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Structured Prediction Helps 3D Human Motion Modelling

Aksan

Kaufmann

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

197

151

View full text Add to dashboard Cite

show abstract

“…Another widely used strategy is to divide the 3D pose estimation task into two decoupled subtasks: 2D pose detection, followed by 3D pose inference from 2D poses. These methods comprise a 2D pose detector and a subsequent optimization [42,41,43] or regression [4,3,16,27,33,13,18,7,11] step to estimate 3D pose . In these methods, the 2D pose and 3D pose estimation stages are separated, making these 3D pose estimators generalize well on outdoor images.…”

Section: Related Workmentioning

confidence: 99%

Not All Parts Are Created Equal: 3D Pose Estimation by Modeling Bi-Directional Dependencies of Body Parts

Wang

Huang

Wang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Not all the human body parts have the same degree of freedom (DOF) due to the physiological structure. For example, the limbs may move more flexibly and freely than the torso does. Most of the existing 3D pose estimation methods, despite the very promising results achieved, treat the body joints equally and consequently often lead to larger reconstruction errors on the limbs. In this paper, we propose a progressive approach that explicitly accounts for the distinct DOFs among the body parts. We model parts with higher DOFs like the elbows, as dependent components of the corresponding parts with lower DOFs like the torso, of which the 3D locations can be more reliably estimated. Meanwhile, the high-DOF parts may in turn impose a constraint on where the low-DOF ones lie. As a result, parts with different DOFs supervise one another, yielding physically constrained and plausible pose-estimation results. To further facilitate the prediction of the high-DOF parts, we introduce a pose-attribute estimation, where the relative location of a limb joint with respect to the torso, which has the least DOF of a human body, is explicitly estimated and further fed to the joint-estimation module. The proposed approach achieves very promising results, outperforming the state of the art on several benchmarks.

show abstract

“…However, the well-known 3d pose datasets [17], [18] contain 3d motion capture (MoCap) data recorded in controlled setup in indoor settings. Hence, 3d supervised learning methods [4], [7], [9], [19] do not generalize well to datasets in the wild where 3d ground-truth is not present.…”

Section: Introductionmentioning

confidence: 99%

“…It is a severely illposed problem which has been formulated in recent literature [3]- [13], [20] as a supervised learning problem given the availability of 3d human motion capture datasets [17], [18]. Most of these works focus on end-to-end 3d pose estimation from single images [3], [5]- [9], [12], [13], [20], while some utilize temporal sequences for estimating 3d pose from video [10], [11]. Our work is related to the problem of 3d pose estimation from a single image, which can be applicable for videos, but without utilizing any temporal information.…”

Section: Introductionmentioning

confidence: 99%

Lifting 2d Human Pose to 3d : A Weakly Supervised Approach

Biswas

Sinha

Gupta

et al. 2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Estimating 3d human pose from monocular images is a challenging problem due to the variety and complexity of human poses and the inherent ambiguity in recovering depth from the single view. Recent deep learning based methods show promising results by using supervised learning on 3d pose annotated datasets. However, the lack of large-scale 3d annotated training data captured under in-the-wild settings makes the 3d pose estimation difficult for in-the-wild poses. Few approaches have utilized training images from both 3d and 2d pose datasets in a weakly-supervised manner for learning 3d poses in unconstrained settings. In this paper, we propose a method which can effectively predict 3d human pose from 2d pose using a deep neural network trained in a weakly-supervised manner on a combination of ground-truth 3d pose and groundtruth 2d pose. Our method uses re-projection error minimization as a constraint to predict the 3d locations of body joints, and this is crucial for training on data where the 3d ground-truth is not present. Since minimizing re-projection error alone may not guarantee an accurate 3d pose, we also use additional geometric constraints on skeleton pose to regularize the pose in 3d. We demonstrate the superior generalization ability of our method by cross-dataset validation on a challenging 3d benchmark dataset MPI-INF-3DHP containing in the wild 3d poses.Index Terms-3d human pose estimation, skeleton, weaklysupervised learning, deep learning.

show abstract

Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Cited by 195 publications

References 45 publications

Structured Prediction Helps 3D Human Motion Modelling

Structured Prediction Helps 3D Human Motion Modelling

Not All Parts Are Created Equal: 3D Pose Estimation by Modeling Bi-Directional Dependencies of Body Parts

Lifting 2d Human Pose to 3d : A Weakly Supervised Approach

Contact Info

Product

Resources

About