First-person pose recognition using egocentric workspaces

Abstract

We tackle the problem of estimating the 3D pose of an individual's upper limbs (arms+hands) from a chest mounted depth-camera. Importantly, we consider pose estimation during everyday interactions with objects. Past work shows that strong pose+viewpoint priors and depth-based features are crucial for robust performance. In egocentric views, hands and arms are observable within a well defined volume in front of the camera. We call this volume an egocentric workspace. A notable property is that hand appearance correlates with workspace location. To exploit this correlation, we classify arm+hand configurations in a global egocentric coordinate frame, rather than a local scanning window. This greatly simplify the architecture and improves performance. We propose an efficient pipeline which 1) generates synthetic workspace exemplars for training using a virtual chest-mounted camera whose intrinsic parameters match our physical camera, 2) computes perspective-aware depth features on this entire volume and 3) recognizes discrete arm+hand pose classes through a sparse multi-class SVM. We achieve state-of-the-art hand pose recognition performance from egocentric RGB-D images in real-time.

Learning Joint Reconstruction of Hands and Manipulated Objects

“…Hand-object reconstruction. Joint reconstruction of hands and objects has been studied with multi-view RGB [2,42,74] and RGB-D input with either optimization [16,17,43,47,62,[69][70][71] or classification [51][52][53][54] approaches. These works use rigid objects, except for a few that use articulated [70] or deformable objects [69].…”

xR-EgoPose: Egocentric 3D Human Pose From an HMD Camera

“…Capturing full 3D body motion from head-mounted cameras is considerably more challenging. Some head-mounted capture systems are based on RGB-D input and reconstruct mostly hand, arm and torso motions [40,57]. Jiang and Grauman [20] reconstruct full body pose from footage taken from a camera worn on the chest by estimating egomotion from the observed scene, but their estimates lack accuracy and have high uncertainty.…”

Analysis of the Hands in Egocentric Vision: A Survey

“…An important difficulty in hand pose estimation lies in object occlusions and self-occlusions that make it hard to localize hidden joints/ parts of the hand. Some authors proposed the use of 3D cameras or depth sensors in conjunction with sensorbased techniques to train hand pose estimators more robust to self-occlusions [17], [97], [79], [69], [98]. However, as discussed above, the use of 3D imaging techniques might not be easily translated to FPV.…”

“…One of the advantages of using depth images for extracting the hand pose is the possibility to synthesize large training sets of realistic depth maps by using computer graphics [17], [97]. In [17], the authors tackled hand pose estimation as a multiclass classification problem by using a hierarchical cascade architecture.…”

“…Compared to other localization tasks, hand pose estimation presents a higher proportion of approaches that use 3D and depth sensors. This choice has several advantages: 1) the possibility to use motion capture methods for automatically obtaining the ground truth joint positions [69], [102]; 2) the availability of multiple streams (color and depth) that can be combined to refine the estimations [79], [106]; and 3) the possibility to synthesize large datasets of realistic depth maps [17], [97]. In the past few years, human pose estimation approaches [103], [104] have been successfully adapted to the egocentric POV, in order to estimate the hand and arm pose from monocular color images [35], [99].…”