Integrating Human Gaze into Attention for Egocentric Activity Recognition

Min, Kyle; Corso, Jason J.

doi:10.1109/wacv48630.2021.00111

Cited by 28 publications

(15 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, in Section 5.5, blind participants suggested enabling future smart glasses to detect specific moments (e.g., sitting in a table or talking with someone) to control feedback delivery. We believe that egocentric activity recognition [59,65,107], a widely-studied problem in the computer vision community, can help realize this experience.…”

Section: Implications Regarding Teachable Interfacesmentioning

confidence: 96%

“…For instance, one recent work proposed an egocentric activity recognition model assuming that input data inherently contains the user's gaze motion [65]. However, that model may not work on data collected by blind people since the assumption, (i.e., eye gaze), only applies to a certain population (i.e., sighted people).…”

Section: Implications Regarding Teachable Interfacesmentioning

confidence: 99%

See 1 more Smart Citation

Accessing Passersby Proxemic Signals through a Head-Worn Camera: Opportunities and Limitations for the Blind

Lee

Sato

Asakawa

et al. 2021

The 23rd International ACM SIGACCESS Conference on Computers and Accessibility

View full text Add to dashboard Cite

Fig. 1. A pedestrian detection scenario demonstrating the data collection in our study. A blind person, wearing smart glasses with our working prototype, and a sighted person walk toward each other in a corridor. The smart glasses detect the sighted passerby, estimate his proximity, and share his relative location and head pose.The spatial behavior of passersby can be critical to blind individuals to initiate interactions, preserve personal space, or practice social distancing during a pandemic. Among other use cases, wearable cameras employing computer vision can be used to extract proxemic signals of others and thus increase access to the spatial behavior of passersby for blind people. Analyzing data collected in a study with blind (N=10) and sighted (N=40) participants, we explore: (i) visual information on approaching passersby captured by a head-worn camera; (ii) pedestrian detection algorithms for extracting proxemic signals such as passerby presence, relative position, distance, and head pose; and (iii) opportunities and limitations of using wearable cameras for helping blind people access proxemics related to nearby people. Our observations and findings provide insights into dyadic behaviors for assistive pedestrian detection and lead to implications for the design of future head-worn cameras and interactions. CCS Concepts: • Human-centered computing → User studies; Empirical studies in HCI; Mobile devices; Empirical studies in accessibility; Accessibility technologies; • Computing methodologies → Computer vision.

show abstract

Section: Implications Regarding Teachable Interfacesmentioning

confidence: 96%

Section: Implications Regarding Teachable Interfacesmentioning

confidence: 99%

Accessing Passersby Proxemic Signals through a Head-Worn Camera: Opportunities and Limitations for the Blind

Lee

Sato

Asakawa

et al. 2021

The 23rd International ACM SIGACCESS Conference on Computers and Accessibility

View full text Add to dashboard Cite

show abstract

“…This leads to a lack of available datasets with cognitively demanding tasks, which could potentially benefit from supplementary gaze information. Consequently, the use of MET systems for EAR has been severely limited and researchers have either explored gaze prediction and modeling (Fathi et al 2012;Huang et al 2020;Li et al 2021;Min and Corso 2021) or utilized non-contextual gaze metrics such as fixation and saccade information for action recognition (Kit and Sullivan 2016;Li et al 2015;Liao et al 2018). Fathi et al (2012) presented a generative probabilistic model that combines gaze prediction and object-based features as multi-modal input for action recognition during daily actions.…”

Section: Gaze-based Action Recognitionmentioning

confidence: 99%

What we see is what we do: a practical Peripheral Vision-Based HMM framework for gaze-enhanced recognition of actions in a medical procedural task

Wang

Kreiner

Lutz

et al. 2023

User Model User-Adap Inter

View full text Add to dashboard Cite

Deep learning models have shown remarkable performances in egocentric video-based action recognition (EAR), but rely heavily on a large quantity of training data. In specific applications with only limited data available, eye movement data may provide additional valuable sensory information to achieve accurate classification performances. However, little is known about the effectiveness of gaze data as a modality for egocentric action recognition. We, therefore, propose the new Peripheral Vision-Based HMM (PVHMM) classification framework, which utilizes context-rich and object-related gaze features for the detection of human action sequences. Gaze information is quantified using two features, the object-of-interest hit and the object–gaze distance, and human action recognition is achieved by employing a hidden Markov model. The classification performance of the framework is tested and validated on a safety-critical medical device handling task sequence involving seven distinct action classes, using 43 mobile eye tracking recordings. The robustness of the approach is evaluated using the addition of Gaussian noise. Finally, the results are then compared to the performance of a VGG-16 model. The gaze-enhanced PVHMM achieves high classification performances in the investigated medical procedure task, surpassing the purely image-based classification model. Consequently, this gaze-enhanced EAR approach shows the potential for the implementation in action sequence-dependent real-world applications, such as surgical training, performance assessment, or medical procedural tasks.

show abstract

“…We also follow [13] and report results for two subsets within val/test: unseen participants and tail classes. For EGTEA, we follow [30,45,47] and report top-1 accuracy and mean class accuracy using the first train/test split.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…First, as there is no audio input to the transformer, we do not use modality encodings either. Second, following previous methods [27,40,45,47,58,59] that train using a single head for actions and report only action accuracy, we use a single summary embedding for actions, rather than verb/noun embeddings. Accordingly, the language model utilises a single word-embedding for actions, with a dimension of 512.…”

Section: F Egtea Implementation Detailsmentioning

confidence: 99%

With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition

Kazakos¹,

Huh²,

Nagrani³

et al. 2021

Preprint

View full text Add to dashboard Cite

In egocentric videos, actions occur in quick succession. We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance. To incorporate the temporal context, we propose a transformer-based multimodal model that ingests video and audio as input modalities, with an explicit language model providing action sequence context to enhance the predictions. We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-the-art performance. Our ablations showcase the advantage of utilising temporal context as well as incorporating audio input modality and language model to rescore predictions. Code and models at: https://github.com/ekazakos/MTCN. IntroductionAction recognition in egocentric video streams from sources like EPIC-KITCHENS poses a number of challenges that differ substantially from those of conventional third-person action recognition -where training and evaluation is on 10 second video clips and classes are quite high-level [31]. Actions are fine-grained (e.g. 'open bottle') and noticeably short, often one second or shorter. Along with the challenge, the footage offers an under-explored opportunity, as actions are captured in long untrimmed videos of well-defined and at-times predictable sequences. For example the action 'wash aubergine' can be part of the following sequence -you first 'take the aubergine', 'turn on the tap', 'wash the aubergine' and finally 'turn off the tap' (Fig. 1). Furthermore, the objects (the aubergine and tap in this case) are persistent over some of the neighbouring actions.

show abstract

Integrating Human Gaze into Attention for Egocentric Activity Recognition

Cited by 28 publications

References 20 publications

Accessing Passersby Proxemic Signals through a Head-Worn Camera: Opportunities and Limitations for the Blind

Accessing Passersby Proxemic Signals through a Head-Worn Camera: Opportunities and Limitations for the Blind

What we see is what we do: a practical Peripheral Vision-Based HMM framework for gaze-enhanced recognition of actions in a medical procedural task

With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition

Contact Info

Product

Resources

About