GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

Mueller, Franziska; Bernard, Florian; Sotnychenko, Oleksandr; Mehta, Dushyant; Sridhar, Srinath; Casas, Dan; Theobalt, Christian

doi:10.1109/cvpr.2018.00013

Cited by 489 publications

(517 citation statements)

References 76 publications

Supporting

Mentioning

512

Contrasting

Unclassified

Order By: Relevance

“…Building on the SynthHands [24] dataset Mueller et al [23] presented the GANerated (GAN) dataset. SynthHands was created by retargeting measured human hand articulation to a rigged meshed model in a mixed reality approach.…”

Section: Considered Datasetsmentioning

confidence: 99%

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Zimmermann

Ceylan

Yang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

384

369

View full text Add to dashboard Cite

Estimating 3D hand pose from single RGB images is a highly ambiguous problem that relies on an unbiased training dataset. In this paper, we analyze cross-dataset generalization when training on existing datasets. We find that approaches perform well on the datasets they are trained on, but do not generalize to other datasets or in-the-wild scenarios. As a consequence, we introduce the first large-scale, multi-view hand dataset that is accompanied by both 3D hand pose and shape annotations. For annotating this realworld dataset, we propose an iterative, semi-automated 'human-in-the-loop' approach, which includes hand fitting optimization to infer both the 3D pose and shape for each sample. We show that methods trained on our dataset consistently perform well when tested on other datasets. Moreover, the dataset allows us to train a network that predicts the full articulated hand shape from a single RGB image. The evaluation set can serve as a benchmark for articulated hand shape estimation. arXiv:1909.04349v3 [cs.CV] 13 Sep 2019 Training Set Evaluation Set Figure 2: Examples from our proposed dataset showing images (top row) and hand shape annotations (bottom row). The training set contains composited images from green screen recordings, whereas the evaluation set contains images recorded indoors and outdoors. The dataset features several subjects as well as object interactions.the key aspects is that we record synchronized images from multiple views, an idea already used previously in [2,27]. The multiple views remove many ambiguities and ease both the manual annotation and automated fitting. The second key aspect of our approach is a semi-automated humanin-the-loop labeling procedure with a strong bootstrapping component. Starting from a sparse set of 2D keypoint annotations (e.g., finger tip annotations) and semi-automatically generated segmentation masks, we propose a hand fitting method that fits a deformable hand model [25] to a set of multi-view input. This fitting yields both 3D hand pose and shape annotation for each view. We then train a multi-view 3D hand pose estimation network using these annotations. This network predicts the 3D hand pose for unlabeled samples in our dataset along with a confidence measure. By verifying confident predictions and annotating least-confident samples in an iterative procedure, we acquire 11592 annotations with moderate manual effort by a human annotator.

show abstract

Section: Considered Datasetsmentioning

confidence: 99%

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Zimmermann

Ceylan

Yang

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

384

369

View full text Add to dashboard Cite

show abstract

“…where dist(θ i , I i ) measures the distance of θ i from the interval of plausible angles I i (as done in [28]), and λ is the hyperparameter that trades off joint limit violations with joint position errors. For our experiments we fix λ = 100.…”

Section: Local Optimisation Methodsmentioning

confidence: 99%

“…The aim then is to find kinematic parameters Θ and hence a kinematically consistent and plausible set of joint positions that achieve a minimum distance to the observed locations, as e.g. done in [26,28]. In this experiment we mimic such a setting by fitting a kinematic skeleton to joint locations that exhibit noise, so that an exact fit may not be possible.…”

Section: Fitting To Noisy Observationsmentioning

confidence: 99%

Convex Optimisation for Inverse Kinematics

Yenamandra

Bernard

Wang

et al. 2019

2019 International Conference on 3D Vision (3DV)

Self Cite

View full text Add to dashboard Cite

We consider the problem of inverse kinematics (IK), where one wants to find the parameters of a given kinematic skeleton that best explain a set of observed 3D joint locations. The kinematic skeleton has a tree structure, where each node is a joint that has an associated geometric transformation that is propagated to all its child nodes. The IK problem has various applications in vision and graphics, for example for tracking or reconstructing articulated objects, such as human hands or bodies. Most commonly, the IK problem is tackled using local optimisation methods. A major downside of these approaches is that, due to the nonconvex nature of the problem, such methods are prone to converge to unwanted local optima and therefore require a good initialisation. In this paper we propose a convex optimisation approach for the IK problem based on semidefinite programming, which admits a polynomial-time algorithm that globally solves (a relaxation of) the IK problem. Experimentally, we demonstrate that the proposed method significantly outperforms local optimisation methods using different real-world skeletons.

show abstract

“…PCK In cases where large errors occur, the value of L pos can be misleading. Hence, following the 3D (hand) pose estimation literature [13,22,28,34], we introduce PCK by computing the percentage of predicted joints lying within a spherical threshold ρ around the target joint position, i.e.…”

Section: Metricsmentioning

confidence: 99%

Structured Prediction Helps 3D Human Motion Modelling

Aksan

Kaufmann

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

197

151

View full text Add to dashboard Cite

Figure 1: We introduce a structured prediction layer (SPL) to the task of 3D human motion modelling. The SP-layer explicitly decomposes the pose into individual joints and can be interfaced with a variety of baseline architectures. We show that on H3.6M and a recent, much larger dataset, AMASS, a variety of baseline models benefit when augmented with an SP-layer. AbstractHuman motion prediction is a challenging and important task in many computer vision application domains. Existing work only implicitly models the spatial structure of the human skeleton. In this paper, we propose a novel approach that decomposes the prediction into individual joints by means of a structured prediction layer that explicitly models the joint dependencies. This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function. The proposed layer is agnostic to the underlying network and can be used with existing architectures for motion modelling. Prior work typically leverages the H3.6M dataset. We show that some state-of-the-art techniques do not perform well when trained and tested on AMASS, a recently released dataset 14 times the size of H3.6M. Our experiments indicate that the proposed layer increases the performance of motion forecasting irrespective of the base network, jointangle representation, and prediction horizon. We furthermore show that the layer also improves motion predictions qualitatively. We make code and models publicly available at https://ait.ethz.ch/projects/2019/spl.

show abstract

GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB

Cited by 489 publications

References 76 publications

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Convex Optimisation for Inverse Kinematics

Structured Prediction Helps 3D Human Motion Modelling

Contact Info

Product

Resources

About