How to Refine 3D Hand Pose Estimation from Unlabelled Depth Data?

Abstract

Data-driven approaches for hand pose estimation from depth images usually require a substantial amount of labelled training data which is quite hard to obtain. In this work, we show how a simple convolutional neural network, pre-trained only on synthetic depth images generated from a single 3D hand model, can be trained to adapt to unlabelled depth images from a real user's hand. We validate our method on two existing and a new dataset that we capture, both quantitatively and qualitatively, demonstrating that we strongly compare to state-of-the-art methods. Additionally, this method can be seen as an extension to existing methods trained on limited datasets, which helps on boosting their performance on new ones.

Self-Supervised 3D Hand Pose Estimation Through Training by Fitting

“…We observe that previous methods which parameterize poses with joint angles [6,55] tend to be sensitive to errors in parent node estimation. Further difficulty is introduced if one attempts to solve these angles via regression.…”

“…However, given that accurate 3D annotations are extremely difficult to obtain, a number of works approach the problem with deep generative models to leverage unlabelled data [1,3,21,28,29,36,49]. Synthesizing depth maps ensures accurate annotations and seem to be a promising alternative but methods that rely only on synthesized data suffer from the large domain shift and actually perform much worse than when trained on less accurate real data [31,34,6]. To reduce the domain gap, Rad et al [31] have proposed a domain adaptation method that tries to minimize feature differences from synthesized versus real images.…”

“…Simulated and unsupervised approaches. One line of work [6,34] considers the more challenging setting of not using any manual labels. These models are trained on labeled but synthesized data and unlabeled real data.…”

“…Dibra et al [6] fine-tune a network trained on synthesized depth maps with unlabeled real data by minimizing a differentiable model-fitting error. While the fine-tuning improves the initial network accuracy, current state-of-the-art still outperforms the fine-tuned network by a large margin.…”

“…While the fine-tuning improves the initial network accuracy, current state-of-the-art still outperforms the fine-tuned network by a large margin. Our approach is similar to [6] in that we also initialize the network with synthetic data and fine-tune with a data-fitting error. However, our method has several key differences including the hand model and model parameterization.…”