E²(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

Abstract

Event cameras are novel bio-inspired sensors, which asynchronously capture pixel-level intensity changes in the form of "events". Due to their sensing mechanism, event cameras have little to no motion blur, a very high temporal resolution and require significantly less power and memory than traditional frame-based cameras. These characteristics make them a perfect fit to several real-world applications such as egocentric action recognition on wearable devices, where fast camera motion and limited power challenge traditional vision sensors. However, the ever-growing field of event-based vision has, to date, overlooked the potential of event cameras in such applications. In this paper, we show that event data is a very valuable modality for egocentric action recognition. To do so, we introduce N-EPIC-Kitchens, the first event-based camera extension of the large-scale EPIC-Kitchens dataset. In this context, we propose two strategies: (i) directly processing eventcamera data with traditional video-processing architectures (E 2 (GO)) and (ii) using event-data to distill optical flow information (E 2 (GO)MO). On our proposed benchmark, we show that event data provides a comparable performance to RGB and optical flow, yet without any additional flow computation at deploy time, and an improved performance of up to 4% with respect to RGB only information. The N-EPIC-Kitchens dataset is available at https://github. com/EgocentricVision/N-EPIC-Kitchens.

Trailers12k: Improving Transfer Learning with a Dual Image and Video Transformer for Multi-Label Movie Trailer Genre Classification

“…Similarly, the transferability of Kinetics to other popular action recognition datasets has been studied directly [18] or indirectly to benchmark architectures [47,13]. Kinetics pretraining has also been applied to other action recognition settings, such as egocentric actions [33], action recognition from drones [9] or actions in the dark [52]. Other studies have used Kinetics to initialize models for more distant video tasks, including sign language recognition [25] or autonomous vehicle decision-making [41].…”

Bringing Online Egocentric Action Recognition into the wild

“…Another important aspect to consider when posing real-time constrains is that, in the context of egocentric vision, many techniques attain notable results only by leveraging non-real-time secondary modalities such as the optical flow. Although this modality is highly successful, it has a high computational cost [36], [37], which prevents its use in real-time applications, and increases the size of the model.…”

Hypergraph Neural Network for Skeleton-Based Action Recognition

“…Despite the abundance of conventional frame-like datasets, there is a noticeable scarcity of event-based action recognition datasets. As for the simulated datasets, N-EPIC-Kitchens [75] is an event version of the EPIC-Kitchens generated by the event camera simulator. The event UCF-50 [76] is derived from the UCF-50 action recognition dataset, which was captured by displaying its data on a monitor.…”