Online generative model personalization for hand tracking

Tkach, Anastasia; Tagliasacchi, Andrea; Remelli, Edoardo; Pauly, Mark; Fitzgibbon, Andrew

doi:10.1145/3130800.3130830

Cited by 86 publications

(91 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3D hand shape and pose estimation from depth images: Most previous methods estimate 3D hand shape and pose from depth images by fitting a deformable hand model to the input depth map with iterative optimization [49,30,24,21,51,41]. A recent method [31] was proposed to estimate pose and shape parameters from the depth image using CNNs, and recover 3D hand meshes using LBS.…”

Section: Related Workmentioning

confidence: 99%

3D Hand Shape and Pose Estimation From a Single RGB Image

Ren²,

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

453

430

View full text Add to dashboard Cite

This work addresses a novel and challenging problem of estimating the full 3D hand shape and pose from a single RGB image. Most current methods in 3D hand analysis from monocular RGB images only focus on estimating the 3D locations of hand keypoints, which cannot fully express the 3D shape of hand. In contrast, we propose a Graph Convolutional Neural Network (Graph CNN) based method to reconstruct a full 3D mesh of hand surface that contains richer information of both 3D hand shape and pose. To train networks with full supervision, we create a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. When fine-tuning the networks on real-world datasets without 3D ground truth, we propose a weakly-supervised approach by leveraging the depth map as a weak supervision in training. Through extensive evaluations on our proposed new datasets and two public datasets, we show that our proposed method can produce accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose estimation accuracy when compared with state-of-the-art methods.

show abstract

Section: Related Workmentioning

confidence: 99%

3D Hand Shape and Pose Estimation From a Single RGB Image

Ren²,

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

453

430

View full text Add to dashboard Cite

show abstract

“…In the context of hand shape estimation, the majority of methods fall into the category of model-based techniques. These approaches were developed in a strictly controlled environment and utilize either depth data directly [29,30,33] or use multi-view stereo methods for reconstruction [2]. More related to our work are approaches that fit statistical human shape models to observations [4,21] from in-thewild color images as input.…”

Section: Related Workmentioning

confidence: 99%

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Zimmermann

Ceylan

Yang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

386

369

View full text Add to dashboard Cite

Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this realworld dataset, we propose an iterative, semi-automated 'human-in-the-loop' approach, which includes hand fitting optimization to infer both the 3D pose and shape for each sample. We show that methods trained on our dataset consistently perform well when tested on other datasets. Moreover, the dataset allows us to train a network that predicts the full articulated hand shape from a single RGB image. The evaluation set can serve as a benchmark for articulated hand shape estimation. arXiv:1909.04349v3 [cs.CV] 13 Sep 2019 Training Set Evaluation Set Figure 2: Examples from our proposed dataset showing images (top row) and hand shape annotations (bottom row). The training set contains composited images from green screen recordings, whereas the evaluation set contains images recorded indoors and outdoors. The dataset features several subjects as well as object interactions.the key aspects is that we record synchronized images from multiple views, an idea already used previously in [2,27]. The multiple views remove many ambiguities and ease both the manual annotation and automated fitting. The second key aspect of our approach is a semi-automated humanin-the-loop labeling procedure with a strong bootstrapping component. Starting from a sparse set of 2D keypoint annotations (e.g., finger tip annotations) and semi-automatically generated segmentation masks, we propose a hand fitting method that fits a deformable hand model [25] to a set of multi-view input. This fitting yields both 3D hand pose and shape annotation for each view. We then train a multi-view 3D hand pose estimation network using these annotations. This network predicts the 3D hand pose for unlabeled samples in our dataset along with a confidence measure. By verifying confident predictions and annotating least-confident samples in an iterative procedure, we acquire 11592 annotations with moderate manual effort by a human annotator.

show abstract

“…Using such acceleration mechanisms, real‐time tracking could be achieved by shifting the model adaptation to a prior offline process. Alternatively, a progressive model adaptation scheme could be realized by employing the recent Levenberg–Marquardt Kalman filter [SHA15, TTR*17].…”

Section: Evaluation Discussion and Applicationsmentioning

confidence: 99%

Markerless Multiview Motion Capture with 3D Shape Model Adaptation

Fechteler

Hilsmann

Eisert

2019

Computer Graphics Forum

View full text Add to dashboard Cite

In this paper, we address simultaneous markerless motion and shape capture from 3D input meshes of partial views onto a moving subject. We exploit a computer graphics model based on kinematic skinning as template tracking model. This template model consists of vertices, joints and skinning weights learned a priori from registered full‐body scans, representing true human shape and kinematics‐based shape deformations. Two data‐driven priors are used together with a set of constraints and cues for setting up sufficient correspondences. A Gaussian mixture model‐based pose prior of successive joint configurations is learned to soft‐constrain the attainable pose space to plausible human poses. To make the shape adaptation robust to outliers and non‐visible surface regions and to guide the shape adaptation towards realistically appearing human shapes, we use a mesh‐Laplacian‐based shape prior. Both priors are learned/extracted from the training set of the template model learning phase. The output is a model adapted to the captured subject with respect to shape and kinematic skeleton as well as the animation parameters to resemble the observed movements. With example applications, we demonstrate the benefit of such footage. Experimental evaluations on publicly available datasets show the achieved natural appearance and accuracy.

show abstract

Online generative model personalization for hand tracking

Cited by 86 publications

References 38 publications

3D Hand Shape and Pose Estimation From a Single RGB Image

3D Hand Shape and Pose Estimation From a Single RGB Image

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Markerless Multiview Motion Capture with 3D Shape Model Adaptation

Contact Info

Product

Resources

About