Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

Abstract

We present an approach that uses a multi-camera system to train fine-grained detectors for keypoints that are prone to occlusion, such as the joints of a hand. We call this procedure multiview bootstrapping: first, an initial keypoint detector is used to produce noisy labels in multiple views of the hand. The noisy detections are then triangulated in 3D using multiview geometry or marked as outliers. Finally, the reprojected triangulations are used as new labeled training data to improve the detector. We repeat this process, generating more labeled data in each iteration. We derive a result analytically relating the minimum number of views to achieve target true and false positive rates for a given detector. The method is used to train a hand keypoint detector for single images. The resulting keypoint detector runs in realtime on RGB images and has accuracy comparable to methods that use depth sensors. The single view detector, triangulated over multiple views, enables 3D markerless hand motion capture with complex object interactions.

To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations

“…where y t (p) is the true location andŷ t (p) ∈ Y is the predicted location of keypoint p Another metric, Probability of Correct Keypoints (PCK) [3,35], is also used to evaluate all models. If a predicted keypoint lies inside a sphere (of radius σ ) around the ground truth, the prediction is deemed correct.…”

Realtime Performance‐Driven Physical Simulation for Facial Animation

“…When the face landmark tracking fails, our physical simulation will maintain a natural facial expression, but the system will not be able to match the subject's expression. We note that very recent facial landmarks trackers such as OpenPose [SJMS17] could alleviate this issue. The landmark tracker's results suffer from jittering issues which also affect the final animation.…”

Figurines, a multimodal framework for tangible storytelling

“…The body tracking algorithm uses a modified version of the Realtime Multi-Person Pose Estimation algorithm [10]. Faces and hands are tracked using OpenPose [1,10,11,12]. Using this software, we are able to compute facial landmark, head pose, eyes and facial Action Units.…”

“…The emergence of RGB-D sensors (like the Kinect) and of Fab-Labs with their new widespread technologies (3D printing, laser cutting ...) lets people envision developing lighter and cheaper systems. In parallel, the increasing performance of human perception algorithms thanks to Deep Learning [1,10,11,12] tends to improve human analysis capabilities of interaction systems. Even if interaction systems take advantages of these recent progresses, there are still challenging problems to address.…”