Hands in action: real-time 3D reconstruction of hands in interaction with objects

Abstract

Abstract-This paper presents a method for vision based estimation of the pose of human hands in interaction with objects. Despite the fact that most robotics applications of human hand tracking involve grasping and manipulation of objects, the majority of methods in the literature assume a free hand, isolated from the surrounding environment. Our hand tracking method is non-parametric, performing a nearest neighbor search in a large database (100000 entries) of hand poses with and without grasped objects. The system operates in real time, it is robust to self occlusions, object occlusions and segmentation errors, and provides full hand pose reconstruction from markerless video. Temporal consistency in hand pose is taken into account, without explicitly tracking the hand in the high dimensional pose space.

Inferring hand pose: A comparative study of visual shape features

“…It should however be noted that this parameterization of hand pose displays certain peculiarities [8], [7]. For example, the Euclidean distances between two poses with change in the outermost joints might be larger than the Euclidean distance between two hands rotated 45…”

“…in [6], [7], models the likelihood p(Y | X), not caring about how the states are distributed, and finds the X that maximizes the likelihood for a certain observed Y , using some optimization procedure.…”

Motion Capture of Hands in Action Using Discriminative Salient Points

“…Most of these methods process the frames independently, however temporal consistency can also be easily enforced [24,5]. Although discriminative methods can recover from errors, their accuracy and the type of poses they can handle depend on the training data.…”

“…To assist this kind of tracking [7] proposed to learn a hand pose prior dependent on the type of object being manipulated, exploiting the idea that manipulating similar objects involves similar hand motions. In the context of object grasping, [5] built a database of 10, 000 hand poses with and without grasped objects to recover the hand pose from images using nearest neighbor search. Recently, [6] proposed to track the manipulated object and the hand at the same time to reduce the search space using collision constraints.…”

“…This, in itself, is very challenging due to the many degrees of freedom, as well as due to selfocclusions and similarity between the fingers. Despite these challenges, recent works [4][5][6] have addressed the problem of a hand interacting with an object. While an object may generate additional occlusions, these can be used to constrain the space of possible poses, as in [7,6].…”