Egocentric Prediction of Action Target in 3D

Li, Yiming; Cao, Ziang; Liang, Andrew S.; Liang, Benjamin; Chen, Luoyao; Zhao, Hang; Chen, Feng

doi:10.1109/cvpr52688.2022.02033

Cited by 10 publications

(3 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By mapping actions to specific regions within a scene, this technique enables the understanding and prediction of human activities in a given environment. Li et al (2022) focused on anticipating as early as possible the target location of a person's object manipulation action in a 3D workspace. While this is a special case of trajectory forecasting, the latter is infeasible in manipulation scenarios and the hands often are located outside the field of view.…”

Section: D Scene Understandingmentioning

confidence: 99%

“…It comprises more than 500K synchronised RGBD frames and gravity directions captured from an egocentric viewpoint with diverse daily activities, for a total of 16 h RGBD recording. EgoPAT3D (Li et al, 2022) is a large multimodality dataset of more than a mil-lion frames of RGB-D and IMU streams, which has been designed for the task of anticipating the target location of a person's object manipulation action in a 3D workspace. The total collection contains 150 recordings, 15 household scene point clouds, 15,000 hand-object actions, 600 min of raw RGB-D/IMU data, 0.9 million hand-object action frames, and 1 million RGB-D frames for the entire dataset.…”

Section: D Scene Understandingmentioning

confidence: 99%

See 1 more Smart Citation

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

What will the future be? We wonder! In this survey, we explore the gap between current research in egocentric vision and the ever-anticipated future, where wearable computing, with outward facing cameras and digital overlays, is expected to be integrated in our every day lives. To understand this gap, the article starts by envisaging the future through character-based stories, showcasing through examples the limitations of current technology. We then provide a mapping between this future and previously defined research tasks. For each task, we survey its seminal works, current state-of-the-art methodologies and available datasets, then reflect on shortcomings that limit its applicability to future research. Note that this survey focuses on software models for egocentric vision, independent of any specific hardware. The paper concludes with recommendations for areas of immediate explorations so as to unlock our path to the future always-on, personalised and life-enhancing egocentric vision.

show abstract

Section: D Scene Understandingmentioning

confidence: 99%

Section: D Scene Understandingmentioning

confidence: 99%

An Outlook into the Future of Egocentric Vision

Plizzari,

Goletto,

Furnari

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

show abstract

“…Recent advances in egocentric action recognition, anticipation, and retrieval focus on building powerful clip-based video models that operate on video clips of a few seconds at a time [12,16,18,25,43,44,54,55]. Despite encouraging performance, these models typically process denselysampled frames with temporally-aware operations, making them computationally heavy.…”

Section: Introductionmentioning

confidence: 99%

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Tan¹,

Nagarajan²,

Grauman³

2023

Preprint

View full text Add to dashboard Cite

Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillationbased approach that learns to reconstruct heavy egocentric video clip features by combining the semantics from a sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel selfsupervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring 200× fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPIC-Kitchens datasets, where our method outperforms state-ofthe-art efficient video understanding methods.

show abstract