Egocentric Object Recognition Leveraging the 3D Shape of the Grasping Hand

Lin, Yizhou; Hua, Gang; Mordohai, Philippos

doi:10.1007/978-3-319-16199-0_52

Cited by 4 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another well-studied problem is hand pose estimation in 3D vision [9, 31], with some work starting to consider first-person data. For instance, Lin et al [22] show that the 3D shape of a grasping hand can help improve recognition of the objects being grasped. We also study pose in the context of interacting with (and thus grasping) objects, but we do not require depth information.…”

Section: Related Workmentioning

confidence: 99%

“…While simply detecting hands may be sufficient for some applications, pixelwise segmentation is often more useful, especially for applications like hand pose recognition and in-hand object detection [22]. Once we have accurately localized hands using the above approach, segmentation is relatively straightforward, as we show in this section.…”

Section: Segmenting Handsmentioning

confidence: 99%

See 1 more Smart Citation

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Bambach

Lee

Crandall

et al. 2015

2015 IEEE International Conference on Computer Vision (ICCV)

364

337

View full text Add to dashboard Cite

Hands appear very often in egocentric video, and their appearance and pose give important cues about what people are doing and what they are paying attention to. But existing work in hand detection has made strong assumptions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Convolutional Neural Networks, and introduce a simple candidate region generation approach that outperforms existing techniques at a fraction of the computational cost. We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an application, we investigate the extent to which hand segmentation alone can distinguish between different activities. We evaluate these techniques on a new dataset of 48 first-person videos of people interacting in realistic environments, with pixel-level ground truth for over 15,000 hand instances.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Segmenting Handsmentioning

confidence: 99%

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Bambach

Lee

Crandall

et al. 2015

2015 IEEE International Conference on Computer Vision (ICCV)

364

337

View full text Add to dashboard Cite

show abstract

“…Thus, the intrinsic characteristics of egocentric vision allowed developing automated methods to study and recognize different grasp types, saving a huge amount of manual labor. Although in most cases the hand grasp analysis has been addressed in a supervised manner (grasp recognition -Section 4.1.1) [68], [87], [118], [119], [120], [121], some authors proposed to tackle this problem using clustering approaches, in order to discover dominant modes of hand-object interaction and identify high-level relationships among clusters (grasp clustering and abstraction -Section 4.1.2) [87], [111], [118], [122].…”

Section: Hand Grasp Analysis and Gesture Recognitionmentioning

confidence: 99%

Analysis of the Hands in Egocentric Vision: A Survey

Bandini

Zariffa

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Egocentric vision (a.k.a. first-person vision -FPV) applications have thrived over the past few years, thanks to the availability of affordable wearable cameras and large annotated datasets. The position of the wearable camera (usually mounted on the head) allows recording exactly what the camera wearers have in front of them, in particular hands and manipulated objects. This intrinsic advantage enables the study of the hands from multiple perspectives: localizing hands and their parts within the images; understanding what actions and activities the hands are involved in; and developing human-computer interfaces that rely on hand gestures. In this survey, we review the literature that focuses on the hands using egocentric vision, categorizing the existing approaches into: localization (where are the hands or part of them?); interpretation (what are the hands doing?); and application (e.g., systems that used egocentric hand cues for solving a specific problem). Moreover, a list of the most prominent datasets with hand-based annotations is provided.

show abstract

“…With development of portable depth sensors, some work that has been done with RGBD images shows promising results in the field of egocentric object recognition [12] [24] and activity recognition [16] [20]. Rogez et al [18] [19] focused on the task of 3D hand pose detection and trained the detector with depth information using synthetic training exemplars.…”

Section: Previous Workmentioning

confidence: 99%

“…[16] [24] further showed that foreground segmentation is efficient enough to separate the hand and handled object area by combining RGB image with depth data. Lin et al [12] built a generic skin-tone model with histogram of oriented normal vectors (HONV) features that works well with 3D point cloud. Useful as the depth information is, current portable RGB-D sensors, like stereo cameras and cameras with structured light, have drawbacks on large power consumption.…”

Section: Previous Workmentioning

confidence: 99%

Gesture-based Bootstrapping for Egocentric Hand Segmentation

Zhang,

Boddeti,

Kitani

2016

Preprint

View full text Add to dashboard Cite

Accurately identifying hands in images is a key subtask for human activity understanding with wearable firstperson point-of-view cameras. Traditional hand segmentation approaches rely on a large corpus of manually labeled data to generate robust hand detectors. However, these approaches still face challenges as the appearance of the hand varies greatly across users, tasks, environments or illumination conditions. A key observation in the case of many wearable applications and interfaces is that, it is only necessary to accurately detect the user's hands in a specific situational context. Based on this observation, we introduce an interactive approach to learn a person-specific hand segmentation model that does not require any manually labeled training data. The training in our approach proceeds in two steps, an interactive bootstrapping step for identifying moving hand regions, followed by learning a personalized user specific hand appearance model. Concretely, our approach uses two convolutional neural networks at training stage:(1) a trained gesture network that uses pre-defined motion information to detect the hand regions which are delivered to the next network for training the hand detector; and (2) an appearance network that learns a person specific model of the hand region based on the output of the gesture network. During training, to make the appearance network robust to errors in the gesture network, the loss function of the former network incorporates the confidence of the gesture network while learning. Experiments demonstrate the robustness of our approach with an F 1 score over 0.8 on all challenging datasets across a wide range of illumination and hand appearance variations, improving upon a baseline approach by more than 10%.

show abstract

Egocentric Object Recognition Leveraging the 3D Shape of the Grasping Hand

Cited by 4 publications

References 27 publications

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Analysis of the Hands in Egocentric Vision: A Survey

Gesture-based Bootstrapping for Egocentric Hand Segmentation

Contact Info

Product

Resources

About