In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Li, Yin; Liu, Miao; Rehg, James M.

doi:10.1007/978-3-030-01228-1_38

Cited by 244 publications

(330 citation statements)

References 78 publications

Supporting

Mentioning

329

Contrasting

Order By: Relevance

“…Although the type of information and ground truth annotations made available by the authors is heterogeneous, it is possible to identify some sub-areas that are more recurrent than others. The vast majority of datasets provided hand segmentation masks [2], [127], [128], [16], [22], [135], [46], [154], [47], [49], reflecting the high number of approaches proposed in this area (Section 3). However, the high number of datasets is counterbalanced by a relative low number of annotated frames, usually in the order of a few hundreds or thousands of images.…”

Section: Fpv Datasets With Hand Annotationmentioning

confidence: 99%

“…To expedite the lenghty pixel-level annotation process and build larger datasets for hand segmentation, some authors proposed semi-automated techniques, for example based on Grabcut [135], [43]. Actions/activities [150], [67], [135], [98], [154], [47] and hand gestures [127], [128], [107], [94], [109], [152], [126] are other common information that were captured and annotated in many datasets. This large amount of data has been used by researchers for developing robust HCI applications that relied on hand gestures.…”

Section: Fpv Datasets With Hand Annotationmentioning

confidence: 99%

See 1 more Smart Citation

Analysis of the Hands in Egocentric Vision: A Survey

Bandini

Zariffa

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Egocentric vision (a.k.a. first-person vision -FPV) applications have thrived over the past few years, thanks to the availability of affordable wearable cameras and large annotated datasets. The position of the wearable camera (usually mounted on the head) allows recording exactly what the camera wearers have in front of them, in particular hands and manipulated objects. This intrinsic advantage enables the study of the hands from multiple perspectives: localizing hands and their parts within the images; understanding what actions and activities the hands are involved in; and developing human-computer interfaces that rely on hand gestures. In this survey, we review the literature that focuses on the hands using egocentric vision, categorizing the existing approaches into: localization (where are the hands or part of them?); interpretation (what are the hands doing?); and application (e.g., systems that used egocentric hand cues for solving a specific problem). Moreover, a list of the most prominent datasets with hand-based annotations is provided.

show abstract

Section: Fpv Datasets With Hand Annotationmentioning

confidence: 99%

Section: Fpv Datasets With Hand Annotationmentioning

confidence: 99%

Analysis of the Hands in Egocentric Vision: A Survey

Bandini

Zariffa

2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…EGTEA Gaze++ is a recently collected dataset with approximately 10K samples of 106 activity classes. We use the first split as [14], which contains 8299 training and 2022 testing instances.…”

Section: Egocentric Video Datasetsmentioning

confidence: 99%

Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

Luo

Yuille

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

169

View full text Add to dashboard Cite

Temporal reasoning is an important aspect of video analysis. 3D CNN shows good performance by exploring spatial-temporal features jointly in an unconstrained way, but it also increases the computational cost a lot. Previous works try to reduce the complexity by decoupling the spatial and temporal filters. In this paper, we propose a novel decomposition method that decomposes the feature channels into spatial and temporal groups in parallel. This decomposition can make two groups focus on static and dynamic cues separately. We call this grouped spatial-temporal aggregation (GST). This decomposition is more parameterefficient and enables us to quantitatively analyze the contributions of spatial and temporal features in different layers. We verify our model on several action recognition tasks that require temporal reasoning and show its effectiveness.

show abstract

“…A way to circumvent this is to leverage rich human activity datasets that contain task-labeled images and videos of humans manipulating objects [5]- [7]. The caveat is that these datasets are often 2D and they lack annotations that could facilitate the inference of a 6D grasp pose which is Mia Toyota Research Institute ("TRI") provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.…”

Section: Introductionmentioning

confidence: 99%

Learning Task-Oriented Grasping From Human Activity Datasets

Kokić¹,

Kragić²,

Bohg

2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

We propose to leverage a real-world, human activity RGB datasets to teach a robot Task-Oriented Grasping (TOG). On the one hand, RGB-D datasets that contain hands and objects in interaction often lack annotations due to the manual effort in obtaining them. On the other hand, RGB datasets are often annotated with labels that do not provide enough information to infer a 6D robotic grasp pose. However, they contain examples of grasps on a variety of objects for many different tasks. Thereby, they provide a much richer source of supervision than RGB-D datasets. We propose a model that takes as input an RGB image and outputs a hand pose and configuration as well as an object pose and a shape. We follow the insight that jointly estimating hand and object poses increases accuracy compared to estimating these quantities independently of each other. Quantitative experiments show that training an object pose predictor with the hand pose information (and vice versa) is better than training without this information. Given the trained model, we process an RGB dataset to automatically obtain training data for a TOG model. This model takes as input an object point cloud and a task and outputs a suitable region for grasping, given the task. Qualitative experiments show that our model can successfully process a real-world dataset. Experiments with a robot demonstrate that this data allows a robot to learn task-oriented grasping on novel objects.

show abstract

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

Cited by 244 publications

References 78 publications

Analysis of the Hands in Egocentric Vision: A Survey

Analysis of the Hands in Egocentric Vision: A Survey

Grouped Spatial-Temporal Aggregation for Efficient Action Recognition

Learning Task-Oriented Grasping From Human Activity Datasets

Contact Info

Product

Resources

About