SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos

Osman, Nada; Camporese, Guglielmo; Coscia, Pasquale; Ballan, Lamberto

doi:10.1109/iccvw54120.2021.00383

Cited by 15 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Near-term anticipation involves predicting label for the immediate next action that would occur in the range of a few seconds having observed a short video segment of duration of a few seconds. Prior work propose a variety of temporal modeling techniques to encode the observed segment such as regression networks [63], reinforced encoder-decoder network [19], TCNs [67], temporal segment network [12], LSTMs [16,17,49], VAEs [44,65] and transformers [21].…”

Section: Related Workmentioning

confidence: 99%

Rethinking Learning Approaches for Long-Term Action Anticipation

Nawhal

Jyothi

Mori

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Action anticipation involves predicting future actions having observed the initial portion of a video. Typically, the observed video is processed as a whole to obtain a video-level representation of the ongoing activity in the video, which is then used for future prediction. We introduce Anticipatr which performs long-term action anticipation leveraging segment-level representations learned using individual segments from different activities, in addition to a video-level representation. We propose a two-stage learning approach to train a novel transformer-based model that uses these two types of representations to directly predict a set of future action instances over any given anticipation duration. Results on Breakfast, 50Salads, Epic-Kitchens-55, and EGTEA Gaze+ datasets demonstrate the effectiveness of our approach.

show abstract

Section: Related Workmentioning

confidence: 99%

Rethinking Learning Approaches for Long-Term Action Anticipation

Nawhal

Jyothi

Mori

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In particular, they anticipate future actions considering videos acquired from the point of view of a robot which interacts with a human. A series of works [33,34,68] address the task considering the LSTM networks to encode the features related to the past. The authors of [79] focused on the goal representation to predict the next action from the first person view.…”

Section: Action Anticipationmentioning

confidence: 99%

“…Furthermore, anticipating what a worker will do and which objects he will interact with provides information to improve safety in a factory, for example by notifying the user with an alert in case a dangerous action or interaction is anticipated. Many recent works investigated human behavior understanding considering different tasks such as action recognition [28,85,30,103,48,59], object detection [38,37,77,76], human-object interaction detection [39,43,82,66], action anticipation [31,35,33,68] as well as the detection of the next active objects [3,32,46,41].…”

Section: Introductionmentioning

confidence: 99%

MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain

Ragusa¹,

Furnari²,

Farinella³

2022

Preprint

View full text Add to dashboard Cite

Wearable cameras allow to acquire images and videos from the user's perspective. These data can be processed to understand humans behavior. Despite human behavior analysis has been thoroughly investigated in third person vision, it is still understudied in egocentric settings and in particular in industrial scenarios. To encourage research in this field, we present MECCANO, a multimodal dataset of egocentric videos to study humans behavior understanding in industrial-like settings. The multimodality is characterized by the presence of gaze signals, depth maps and RGB videos acquired simultaneously with a custom headset. The dataset has been explicitly labeled for fundamental tasks in the context of human behavior understanding from a first person view, such as recognizing and anticipating human-object interactions. With the MECCANO dataset, we explored five different tasks including 1) Action Recognition, 2) Active Objects Detection and Recognition, 3) Egocentric Human-Objects Interaction Detection, 4) Action Anticipation and 5) Next-Active Objects Detection. We propose a benchmark aimed to study human behavior in the considered industrial-like scenario which demonstrates that the investigated tasks and the considered scenario are challenging for state-of-the-art algorithms. To support research in this field, we publicy release the dataset at https://iplab.dmi.unict.it/MECCANO/.

show abstract

“…This is a challenging task, as the same action can be performed in many different ways, and can be affected by factors such as the viewpoint, lighting conditions, and the presence of other objects in the scene. Despite these challenges, action recognition has many potential applications in fields such as video surveillance [ 1 , 2 ], sports analysis [ 3 , 4 , 5 ], and human–computer interaction [ 6 , 7 , 8 ].…”

Section: Introductionmentioning

confidence: 99%

Joint-Based Action Progress Prediction

Pucci

Becattini

Bimbo

2023

Sensors

View full text Add to dashboard Cite

Action understanding is a fundamental computer vision branch for several applications, ranging from surveillance to robotics. Most works deal with localizing and recognizing the action in both time and space, without providing a characterization of its evolution. Recent works have addressed the prediction of action progress, which is an estimate of how far the action has advanced as it is performed. In this paper, we propose to predict action progress using a different modality compared to previous methods: body joints. Human body joints carry very precise information about human poses, which we believe are a much more lightweight and effective way of characterizing actions and therefore their execution. Estimating action progress can in fact be determined based on the understanding of how key poses follow each other during the development of an activity. We show how an action progress prediction model can exploit body joints and integrate it with modules providing keypoint and action information in order to be run directly from raw pixels. The proposed method is experimentally validated on the Penn Action Dataset.

show abstract

SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos

Cited by 15 publications

References 11 publications

Rethinking Learning Approaches for Long-Term Action Anticipation

Rethinking Learning Approaches for Long-Term Action Anticipation

MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain

Joint-Based Action Progress Prediction

Contact Info

Product

Resources

About