Hand parsing for fine-grained recognition of human grasps in monocular images

Abstract

We propose a novel method for performing finegrained recognition of human hand grasp types using a single monocular image to allow computational systems to better understand human hand use. In particular, we focus on recognizing challenging grasp categories which differ only by subtle variations in finger configurations. While much of the prior work on understanding human hand grasps has been based on manual detection of grasps in video, this is the first work to automate the analysis process for fine-grained grasp classification. Instead of attempting to utilize a parametric model of the hand, we propose a hand parsing framework which leverages a data-driven learning to generate a pixelwise segmentation of a hand into finger and palm regions. The proposed approach makes use of appearance-based cues such as finger texture and hand shape to accurately determine hand parts. We then build on the hand parsing result to compute high-level grasp features to learn a supervised fine-grained grasp classifier. To validate our approach, we introduce a grasp dataset recorded with a wearable camera, where the hand and its parts have been manually segmented with pixel-wise accuracy. Our results show that our proposed automatic hand parsing technique can improve grasp classification accuracy by over 30 percentage points over a state-of-the-art grasp recognition technique.

Understanding Hand-Object Manipulation with Grasp Types and Object Attributes

“…Yang et al [32] utilized a convolutional neural network to classify hand grasp types on unstructured public dataset and presented the usefulness of grasp types for predicting action intention. Saran et al [28] used detected hand parts as intermediate representation to recognize fine-grained grasp types. However, the recognition performance is still not good enough for practical usage in real-world environments.…”

Graph-Based Hand-Object Meshes and Poses Reconstruction With Multi-Modal Input

“…One stream of works in this direction focuses on a hand alone, e.g. hand pose inferring and RGB or RGBD data configuration to either control the hand [13][14][15] or infer manipulation behavior [16][17][18]. Another line of study incorporates the concept that an object's structure determines the hand pose and investigates the hand along with the object [19][20][21].…”

Future Person Localization in First-Person Videos

“…A typical problem setting involving first-person vision is to recognize activities of camera wearers. Recently, some work has focused on activity recognition [7,22,23,28], activity forecasting [6,9,26,31], person identification [11], gaze anticipation [45] and grasp recognition [3,4,21,35]. Similar to our setting, other work has also tried to recognize behaviors of other people observed in first-person videos, e.g., group discovery [2], eye contact detection [42] and activity recognition [33,34,44].…”