Efficient 3D Hand Tracking in Articulation Subspaces for the Manipulation of Virtual Objects

Abstract

We propose an efficient method for model-based 3D tracking of hand articulations observed from an egocentric viewpoint that aims at supporting the manipulation of virtual objects. Previous modelbased approaches optimize non-convex objective functions defined in the 26 Degrees of Freedom (DoFs) space of possible hand articulations. In our work, we decompose this space into six articulation subspaces (6 DoFs for the palm and 4 DoFs for each finger). We also label each finger with a Gaussian model that is propagated between successive image frames. As confirmed by a number of experiments, this divide-and-conquer approach tracks hand articulations more accurately than existing model-based approaches. At the same time, real time performance is achieved without the need of GPGPU processing. Additional experiments show that the proposed approach is preferable for supporting the accurate manipulation of virtual objects in VR/AR scenarios.

Interlinked Visual Tracking and Robotic Manipulation of Articulated Objects

“…Tracking of articulated systems, such as hands [3], human bodies [4] and robotic structures [5] is an active area of research in computer vision. However, these methods present serious drawbacks for feedback control: in general, they are (i) computationally demanding, (ii) not validated against groundtruth measurements (only superposed skeleton on image rendering), (iii) can experience convergence issues, (iv) some have jumps or jerkiness in frame-to-frame estimation, (v) some assume fixed cameras, and (vi) many are tailored for a specific item (e.g.…”

Visual estimation of articulated objects configuration during manipulation with a humanoid

“…Multiple hypothesis using derivative-or gradient-free optimization techniques have been studied in the line of works (e.g. [6][7] [8]). Such approaches are quite helpful to avoid local minima in optimization.…”