Video-based hand manipulation capture through composite motion control

Wang, Yangang; Min, Jianyuan; Zhang, Jianjie; Liu, Yebin; Xu, Feng; Dai, Qionghai; Chai, Jinxiang

doi:10.1145/2461912.2462000

Cited by 83 publications

(33 citation statements)

References 31 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditionally, hand tracking relied on 2D models but pose variability and occlusions have limited general applicability (see [12,46] for surveys). Recently, sophisticated methods using articulated 3D models have demonstrated high fidelity single and two-handed tracking and even physical object interactions [3,9,10,42,64]. However, these techniques are non-real time making them unsuitable for many interactive applications.…”

Section: Related Workmentioning

confidence: 99%

ThirdLight

Kim

Han

Lim

et al. 2017

Proceedings of the 14th European Conference on Visual Media Production (CVMP 2017)

View full text Add to dashboard Cite

Figure 1: ThirdLight tracks the 3D position of low-cost photosensor markers embedded into arbitrary objects. In this example, a glove embedded with such sensors (circled red) is illuminated at high speed using a low-cost illumination device comprising multiple LEDs and Gray-code masks (circled green). The illumination device triangulates the 3D position of each marker at high speeds (333Hz). Each marker is uniquely identified and in this example used to reconstruct the pose of the user's hand. Our ThirdLight tracker is general purpose and can also be used to recover the pose of heads-up displays, props and controllers for virtual reality and robot manipulation scenarios. ABSTRACTWe present a low-cost 3D tracking system for virtual reality, gesture modeling, and robot manipulation applications which require fast and precise localization of headsets, data gloves, props, or controllers. Our system removes the need for cameras or projectors for sensing, and instead uses cheap LEDs and printed masks for illumination, and low-cost photosensitive markers. The illumination device transmits a spatiotemporal pattern as a series of binary Gray-code patterns. Multiple illumination devices can be combined to localize each marker in 3D at high speed (333Hz). Our method has strengths in accuracy, speed, cost, ambient performance, large working space (1m-5m) and robustness to noise compared with conventional techniques. We compare with a state-of-the-art instrumented glove and vision-based systems to demonstrate the accuracy, scalability, and robustness of our approach. We propose a fast and accurate method for hand gesture modeling using an inverse kinematics approach with the six photosensitive markers. We additionally propose a passive markers system and demonstrate various interaction scenarios as practical applications.

show abstract

Section: Related Workmentioning

confidence: 99%

ThirdLight

Kim

Han

Lim

et al. 2017

Proceedings of the 14th European Conference on Visual Media Production (CVMP 2017)

View full text Add to dashboard Cite

show abstract

“…Previous works on the hand in computer graphics have focused on geometry acquisition and retargeting of the whole hand [Kurihara and Miyata 2004;Li et al 2007;Huang et al 2011], musculotendon simulation based on kinematic muscle paths and moment arms [Albrecht et al 2003;Tsang et al 2005], and grasping and interaction with the environment [ElKoura and Singh 2003;Kry and Pai 2006;Pollard and Zordan 2005;Liu 2008;Liu 2009;Zhao et al 2013;Wang et al 2013]. In our work, we focus on the movement of the fingers and not of the hand and the forearm.…”

Section: Musculotendon Simulatormentioning

confidence: 99%

Biomechanical simulation and control of hands and tendinous systems

et al. 2015

View full text Add to dashboard Cite

Figure 1: Our biomechanical simulation and control framework can model the human hand performing tasks such as writing (a-b), and typing on a keyboard (c). We can also simulate clinical conditions such as boutonniére deformity (d) by cutting a tendon insertion. AbstractThe tendons of the hand and other biomechanical systems form a complex network of sheaths, pulleys, and branches. By modeling these anatomical structures, we obtain realistic simulations of coordination and dynamics that were previously not possible. First, we introduce Eulerian-on-Lagrangian discretization of tendon strands, with a new selective quasistatic formulation that eliminates unnecessary degrees of freedom in the longitudinal direction, while maintaining the dynamic behavior in transverse directions. This formulation also allows us to take larger time steps. Second, we introduce two control methods for biomechanical systems: first, a general-purpose learning-based approach requiring no previous system knowledge, and a second approach using data extracted from the simulator. We use various examples to compare the performance of these controllers.

show abstract

“…They achieved lower posture estimation errors than those of Oikonomidis et al [4], but thus far their approach is not real-time. Wang et al [10] realized motion capture of hand grasping and manipulation data by simultaneously modeling hand articulation, object movement, and interactions between the two in an optimization framework. They obtained physically accurate results, but their method is also not real-time.…”

Section: Related Workmentioning

confidence: 99%

Real-time hand tracking using synergistic inverse kinematics

Schröder¹,

Maycock

Ritter

et al. 2014

2014 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

We present a method for real-time bare hand tracking that utilizes natural hand synergies to reduce the complexity and improve the plausibility of the hand posture estimation. The hand pose and posture are estimated by fitting a virtual hand model to the 3D point cloud obtained from a Kinect camera using an inverse kinematics approach. We use real human hand movements captured with a Vicon motion tracking system as the ground truth for deriving natural hand synergies based on principal component analysis. These synergies are integrated in the tracking scheme by optimizing the posture in a reduced parameter space. Tracking in this reduced space combined with joint limit avoidance constrains the posture estimation to natural hand articulations. The information loss associated with dimension reduction can be dealt with by employing a hierarchical optimization scheme. We show that our synergistic hand tracking approach improves runtime performance and increases the quality of the posture estimation. I. INTRODUCTIONTracking the complete articulation of a freely moving hand is a problem that is an ongoing research topic and has numerous applications in robotics. Many existing hand tracking solutions either require the user to wear cumbersome equipment, are expensive or are inadequate for real-time tracking. Some methods that use consumer-level depth sensors can detect the positions of individual fingers and provide a means for rough gesture interaction, but do not accurately reconstruct the user's hand posture with full degrees of freedom (DoFs).We have built a hand tracking system that uses a Kinect camera to estimate the full articulation of a user's bare hand in real-time. Our method is a generative approach that is based on fitting a virtual hand model to the 3D point cloud obtained from the Kinect sensor's depth camera. We estimate the hand articulation by finding the pose and posture parameters that minimize the error between the observed point cloud and the model surface using inverse kinematics. In doing so, we find the deformation of the 3D hand model that best approximates the observed state of the user's hand.A prevalent issue in tracking a highly articulated object like a hand is the number of DoFs that must be optimized. The analysis of hand synergies aims to identify high-level relationships in hand articulation in order to sensibly reduce the dimensionality of hand posture representations. We obtain such hand synergies through the principal component analysis of motion capture data and use them directly in the tracking process to reduce the parameter space and to naturally constrain the hand posture estimation. This paper extends our recent workshop paper [1] in several aspects: First, we speed up the inverse kinematics

show abstract

Video-based hand manipulation capture through composite motion control

Cited by 83 publications

References 31 publications

ThirdLight

ThirdLight

Biomechanical simulation and control of hands and tendinous systems

Real-time hand tracking using synergistic inverse kinematics

Contact Info

Product

Resources

About