Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

Tekin, Bugra; Márquez-Neila, Pablo; Salzmann, Mathieu; Fua, Pascal

doi:10.1109/iccv.2017.425

Cited by 226 publications

(130 citation statements)

References 69 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Recently, deep architectures have been used to learn 3D representations from RGB images [69,57,37,56,38,46] thanks to the availability of high precise 3D data [24], and are now able to surpass depth-sensors [39]. Chen and Ramanan [11] divided the problem of 3D pose estimation into two parts.…”

Section: Pose Estimationmentioning

confidence: 99%

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Luvizon

Picard

Tabia

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Human pose estimation and action recognition are related tasks since both problems are strongly dependent on the human body representation and analysis. Nonetheless, most recent methods in the literature handle the two problems separately. In this work, we propose a multi-task framework for jointly estimating 2D or 3D human poses from monocular color images and classifying human actions from video sequences. We show that a single architecture can be used to solve both problems in an efficient way and still achieves state-of-the-art or comparable results at each task while running with a throughput of more than 100 frames per second. The proposed method benefits from high parameters sharing between the two tasks by unifying still images and video clips processing in a single pipeline, allowing the model to be trained with data from different categories simultaneously and in a seamlessly way. Additionally, we provide important insights for end-to-end training the proposed multi-task model by decoupling key prediction parts, which consistently leads to better accuracy on both tasks. The reported results on four datasets (MPII, Human3.6M, Penn Action and NTU RGB+D) demonstrate the effectiveness of our method on the targeted tasks. Our source code and trained weights are publicly available at

show abstract

Section: Pose Estimationmentioning

confidence: 99%

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Luvizon

Picard

Tabia

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Our work is also related to 3D pose estimation methods that try to recover 3D locations of joints from 2D images or directly from 3D point cloud and volumetric data (see also [30,50] for related surveys). Most recent methods use deep architectures to extract joints for humans [48,19,41,33,38,75,61,42], hands [15,37,20,63,15,64,14], and more recently some species of animals [43]. However, all these approaches aim to predict a pre-defined set of joints for a particular class of objects.…”

Section: Related Workmentioning

confidence: 99%

Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets

Zhan

Zhou

Kalogerakis

et al. 2019

2019 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

We present a learning method for predicting animation skeletons for input 3D models of articulated characters. In contrast to previous approaches that fit pre-defined skeleton templates or predict fixed sets of joints, our method produces an animation skeleton tailored for the structure and geometry of the input 3D model. Our architecture is based on a stack of hourglass modules trained on a large dataset of 3D rigged characters mined from the web. It operates on the volumetric representation of the input 3D shapes augmented with geometric shape features that provide additional cues for joint and bone locations. Our method also enables intuitive user control of the level-of-detail for the output skeleton. Our evaluation demonstrates that our approach predicts animation skeletons that are much more similar to the ones created by humans compared to several alternatives and baselines.

show abstract

“…Monocular Image Based 3D Pose Estimation. Recently, several methods have been proposed to estimate the 3D pose on the monocular image [11,16,20,28,36,38,40,49].…”

Section: Related Workmentioning

confidence: 99%

Generalizing Monocular 3D Human Pose Estimation in the Wild

Wang

Chen

Guo

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

The availability of the large-scale labeled 3D poses in the Human3.6M dataset plays an important role in advancing the algorithms for 3D human pose estimation from a still image. We observe that recent innovation in this area mainly focuses on new techniques that explicitly address the generalization issue when using this dataset, because this database is constructed in a highly controlled environment with limited human subjects and background variations. Despite such efforts, we can show that the results of the current methods are still error-prone especially when tested against the images taken in-the-wild. In this paper, we aim to tackle this problem from a different perspective. We propose a principled approach to generate high quality 3D pose ground truth given any in-the-wild image with a person inside. We achieve this by first devising a novel stereo inspired neural network to directly map any 2D pose to high quality 3D counterpart. We then perform a carefully designed geometric searching scheme to further refine the joints. Based on this scheme, we build a large-scale dataset with 400,000 in-the-wild images and their corresponding 3D pose ground truth. This enables the training of a high quality neural network model, without specialized training scheme and auxiliary loss function, which performs favorably against the state-of-the-art 3D pose estimation methods. We also evaluate the generalization ability of our model both quantitatively and qualitatively. Results show that our approach convincingly outperforms the previous methods. We make our dataset and code publicly available. 1

show abstract

Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

Cited by 226 publications

References 69 publications

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets

Generalizing Monocular 3D Human Pose Estimation in the Wild

Contact Info

Product

Resources

About