Sphere-meshes for real-time hand modeling and tracking

Abstract

Modern systems for real-time hand tracking rely on a combination of discriminative and generative approaches to robustly recover hand poses. Generative approaches require the specification of a geometric model. In this paper, we propose a the use of sphere-meshes as a novel geometric representation for real-time generative hand tracking. How tightly this model fits a specific user heavily affects tracking precision. We derive an optimization to non-rigidly deform a template model to fit the user data in a number of poses. This optimization jointly captures the user's static and dynamic hand geometry, thus facilitating high-precision registration. At the same time, the limited number of primitives in the tracking template allows us to retain excellent computational performance. We confirm this by embedding our models in an open source real-time registration algorithm to obtain a tracker steadily running at 60Hz. We demonstrate the effectiveness of our solution by qualitatively and quantitatively evaluating tracking precision on a variety of complex motions. We show that the improved tracking accuracy at high frame-rate enables stable tracking of extended and complex motion sequences without the need for per-frame re-initialization. To enable further research in the area of high-precision hand tracking, we publicly release source code and evaluation datasets.

How to Refine 3D Hand Pose Estimation from Unlabelled Depth Data?

“…Schroder et al [20] suggest optimizing in a reduced parameter space and Tagliasacchi et al [24] combine previous results, to show that ICP in combination with temporal, collision, kinematic and data-driven terms can be utilized to track with high robustness and accuracy. Following up on this, Sharp et al [21] enhance this approach utilizing a smooth model and Tkach et al [27] present a new hand model based on sphere meshes. A non-gradient, particle swarm optimization (PSO) approach has been suggested by Oikonomidis et al [17], minimizing "the discrepancy between the appearance and 3D structure of hypothesized instances of a hand model and actual hand observations".…”

“…Next to convolutional data-driven approaches, there have been several generative, model-driven ones that perform iterative optimization. For instance, [13,20,24,26,27] optimize for point cloud correspondences while [17,21,29,18] attempt to find a good pose, by iteratively rendering many synthetic depth images and comparing them to the input image. Such approaches usually perform better on unseen poses, as compared to data-driven ones, when applied to poses quite dissimilar from the ones in training datasets.…”

Hand-Object Interaction Detection with Fully Convolutional Networks

“…The resulting labels are used together with the input point cloud to track the object in real-time. We adapt the model-based hand tracking method of [20,21] towards rigid object tracking. This method is highly efficient, as well as robust against occlusions and noisy RGBD input data.…”

“…Rather than using standard ICP [2], we adapt the robust registration method of [20,21] from articulated tracking to rigid object tracking. This method is robust w.r.t.…”

“…For more details on the correspondence computation and the minimization of the objective function, we refer to [20,21]. Figure 5 shows examples for the model-based object tracking.…”