Online optical marker-based hand tracking with deep labels

Abstract

Fig. 1. Snapshots of our online marker-based hand tracking system on sequences with two-handed and hand-object interactions. We demonstrate a novel marker-labeling and tracking system that enables fully-automatic, real-time estimation of hand poses in challenging interaction scenarios with frequent occlusions. Markers labeled as left hand and right hand are rendered as orange and blue spheres respectively, while markers associated with predefined rigid bodies are rendered as green spheres.Optical marker-based motion capture is the dominant way for obtaining high-fidelity human body animation for special effects, movies, and video games. However, motion capture has seen limited application to the human hand due to the difficulty of automatically identifying (or labeling) identical markers on self-similar fingers. We propose a technique that frames the labeling problem as a keypoint regression problem conducive to a solution using convolutional neural networks. We demonstrate robustness of our labeling solution to occlusion, ghost markers, hand shape, and even motions involving two hands or handheld objects. Our technique is equally applicable to sparse or dense marker sets and can run in real-time to support interaction prototyping with high-fidelity hand tracking and hand presence in virtual reality.

Interactive hand pose estimation using a stretch-sensing soft glove

“…Marker based MoCap approaches (e.g., Vicon [2019]) require multiple, calibrated cameras and, compared to the full-body case, marker occlusions are a more severe problem. In consequence, learning based approaches to marker labelling under occlusion have been proposed [Han et al 2018]. However, the need for multiple cameras restricts the applicability of such approaches.…”

A Three-dimensional Finger Motion Measurement System of a Thumb and an Index Finger Without a Calibration Process

“…Diverse hand motion measurement systems have been actively investigated for the applications of virtual reality (VR), tele-operation, biomechanics, etc. Many methods have been proposed to measure the hand motion; the most common systems are optical marker-based motion capture systems, which use infrared signals to measure the three-dimensional position of markers [1][2][3][4][5]. However, stable and robust measurement of the hand is still challenging due to occluded markers, ghost markers and dense marker sets caused by a high degree of articulation, self-similarity, and the small scale of the hand.…”

Self-Supervised 3D Hand Pose Estimation Through Training by Fitting

“…The improvement in the accuracy of learning-based approaches can be attributed to two factors: choices in network design, and increased amounts of labelled data [29,49,53]. However, acquiring accurate 3D hand pose labels can be extremely difficult; current methods require marker-based motion capture [13], 6DoF sensors [53], or multi-view model-based tracking [45]. In all of these cases, careful supervision and manual cleaning of the labels is additionally necessary for high quality annotations.…”

“…A recent trend in hand pose estimation is to combine the benefits of learning-based discriminative approaches with model-based tracking methods [2,13,33,43]. Model-based tracking casts pose estimation as a frame-wise model-fitting problem and requires no training data.…”

“…Model-based tracking casts pose estimation as a frame-wise model-fitting problem and requires no training data. Learning-based approaches can then be incorporated either by providing better initialization [33,43], or by supporting the fitting with key-point estimations [2,13] to reduce the risk of getting trapped in local minima.…”