Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation Under Hand-Object Interaction

Abstract

We study how well different types of approaches generalise in the task of 3D hand pose estimation under single hand scenarios and handobject interaction. We show that the accuracy of state-of-the-art methods can drop, and that they fail mostly on poses absent from the training set. Unfortunately, since the space of hand poses is highly dimensional, it is inherently not feasible to cover the whole space densely, despite recent efforts in collecting large-scale training datasets. This sampling problem is even more severe when hands are interacting with objects and/or inputs are RGB rather than depth images, as RGB images also vary with lighting conditions and colors. To address these issues, we designed a public challenge (HANDS'19) to evaluate the abilities of current 3D hand pose estimators (HPEs) to interpolate and extrapolate the poses of a training set. More exactly, HANDS'19 is designed (a) to evaluate the influence of both depth and color modalities on 3D hand pose estimation, under the presence or absence of objects; (b) to assess the generalisation abilities w.r.t. four main axes: shapes, articulations, viewpoints, and objects; (c) to explore the use of a synthetic hand models to fill the gaps of current datasets. Through the challenge, the overall accuracy has dramatically improved over the baseline, especially on extrapolation tasks, from 27mm to 13mm mean joint error. Our analyses highlight the impacts of: Data pre-processing, ensemble approaches, the use of a parametric 3D hand model (MANO), and different HPE methods/backbones.

Hand Pose Estimation in the Task of Egocentric Actions

“…Our approach is based on machine learning techniques, namely deep learning. We adopt the approach introduced in [37] as the Voxel-to-Voxel PoseNet and present an ablation study on a recent benchmark dataset HANDS19 Challenge: Task 2 -Depth-Based 3D Hand Pose Estimation while Interacting with Objects [31]. We believe this dataset to be the most challenging depth based egocentric view hand pose estimation dataset available to this date.…”

“…We train and evaluate our system on the data provided in the HANDS19 Challenge: Task 2 -Depth-Based 3D Hand Pose Estimation while Interacting with Objects [31], [65]. This task builds on the F-PHAB dataset [4], where objects are being manipulated by a subject in an egocentric viewpoint, see Figure 4.…”

“…The individual predictions from different frames are first processed by the pose prior approach and the results are then averaged. We [31] represent fraction of frames from test sets that predicted joint locations with maximum error below given threshold. It can be seen that our approach (NTIS) outperforms the others, especially in the domain of lower error thresholds.…”

“…It can be seen that our approach (NTIS) outperforms the others, especially in the domain of lower error thresholds. Other approaches of the challenge (A2J, CrazyHand and BT) are summarized in [31]. also tried to first average the raw predictions from different frames and then using the pose prior.…”

“…• Using temporal context to produce more robust hand pose estimations. • Constituting state-of-the-art results in the HANDS19 Challenge: Task 2 -Depth-Based 3D Hand Pose Estimation while Interacting with Objects [31]. The rest of the paper is divided as follows; Section II deals with related work in the field of hand pose estimation.…”