Understanding Everyday Hands in Action from RGB-D Images

Rogez, Grégory; Supancic, James Steven; Ramanan, Deva

doi:10.1109/iccv.2015.443

Cited by 138 publications

(114 citation statements)

References 36 publications

Supporting

Mentioning

107

Contrasting

Unclassified

Order By: Relevance

“…In this section, we present experiments and results that test the following hypotheses: 1) Using a combination of real and synthetic data to train our CNN for hand-object estimation yields more accurate results in terms of relative poses than reconstructing the handobject pairs. We test this through a qualitative comparison of ours and a state-of-the art method for hand-object reconstruction [15] on the challenging real-world dataset GUN-71 [24] (Sec. VI-A).…”

Section: Methodsmentioning

confidence: 99%

Learning Task-Oriented Grasping From Human Activity Datasets

Kokić¹,

Kragić²,

Bohg

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

We propose to leverage a real-world, human activity RGB datasets to teach a robot Task-Oriented Grasping (TOG). On the one hand, RGB-D datasets that contain hands and objects in interaction often lack annotations due to the manual effort in obtaining them. On the other hand, RGB datasets are often annotated with labels that do not provide enough information to infer a 6D robotic grasp pose. However, they contain examples of grasps on a variety of objects for many different tasks. Thereby, they provide a much richer source of supervision than RGB-D datasets. We propose a model that takes as input an RGB image and outputs a hand pose and configuration as well as an object pose and a shape. We follow the insight that jointly estimating hand and object poses increases accuracy compared to estimating these quantities independently of each other. Quantitative experiments show that training an object pose predictor with the hand pose information (and vice versa) is better than training without this information. Given the trained model, we process an RGB dataset to automatically obtain training data for a TOG model. This model takes as input an object point cloud and a task and outputs a suitable region for grasping, given the task. Qualitative experiments show that our model can successfully process a real-world dataset. Experiments with a robot demonstrate that this data allows a robot to learn task-oriented grasping on novel objects.

show abstract

Section: Methodsmentioning

confidence: 99%

Learning Task-Oriented Grasping From Human Activity Datasets

Kokić¹,

Kragić²,

Bohg

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Some authors used the depth information to perform a background/foreground segmentation followed by hand/object segmentation within the foreground region by using appearance information [67], [68], [69]. Wan et al [67] used a time-of-flight (ToF) camera to capture the scene during hand-object interactions.…”

Section: D Segmentationmentioning

confidence: 99%

“…Thus, after thresholding the histogram of depth values to isolate the foreground, hand pixels were detected by combining color (RGB thresholds) and texture (Gabor filters) features. The same ToF camera (Creative Senz3D TM ) was used by Rogez et al [68]. The authors trained a multi-class classifier on synthetic depth images of 1,500 different hand poses, in order to recognize one of these poses in the test depth images, thus producing a coarse segmentation mask.…”

Section: D Segmentationmentioning

confidence: 99%

Analysis of the Hands in Egocentric Vision: A Survey

Bandini

Zariffa

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Egocentric vision (a.k.a. first-person vision -FPV) applications have thrived over the past few years, thanks to the availability of affordable wearable cameras and large annotated datasets. The position of the wearable camera (usually mounted on the head) allows recording exactly what the camera wearers have in front of them, in particular hands and manipulated objects. This intrinsic advantage enables the study of the hands from multiple perspectives: localizing hands and their parts within the images; understanding what actions and activities the hands are involved in; and developing human-computer interfaces that rely on hand gestures. In this survey, we review the literature that focuses on the hands using egocentric vision, categorizing the existing approaches into: localization (where are the hands or part of them?); interpretation (what are the hands doing?); and application (e.g., systems that used egocentric hand cues for solving a specific problem). Moreover, a list of the most prominent datasets with hand-based annotations is provided.

show abstract

“…Related to this purpose, inexpensive RGB-D camera has greatly enhanced the efficiency of visual sensing. Many state-of-the-art works have explored recognition of human daily activities captured by egocentric vision [7,3,2], most of which are focusing on daily activities such as opening the coffee jar or grasp a mug when preparing coffee or making a cake in kitchen scene [7]. These activities tend to interact with a variety of objects in common living scenes with a restricted manipulating complexity.…”

Section: Introductionmentioning

confidence: 99%

“…One major approach to analyze these egocentric experiences is through hands-objects interactions (HOI). Rogez et al [3] build an RGB-D egocentric dataset in describing fine-grained gasps, which is suggestive of hand pose, hand-object contacting points and contact force vectors greatly contribute to understanding HOI activities. Touch points are certainly strong clues to things been interacted and activities using them.…”

Section: Introductionmentioning

confidence: 99%