With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition

Abstract

In egocentric videos, actions occur in quick succession. We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance. To incorporate the temporal context, we propose a transformer-based multimodal model that ingests video and audio as input modalities, with an explicit language model providing action sequence context to enhance the predictions. We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-the-art performance. Our ablations showcase the advantage of utilising temporal context as well as incorporating audio input modality and language model to rescore predictions. Code and models at: https://github.com/ekazakos/MTCN. IntroductionAction recognition in egocentric video streams from sources like EPIC-KITCHENS poses a number of challenges that differ substantially from those of conventional third-person action recognition -where training and evaluation is on 10 second video clips and classes are quite high-level [31]. Actions are fine-grained (e.g. 'open bottle') and noticeably short, often one second or shorter. Along with the challenge, the footage offers an under-explored opportunity, as actions are captured in long untrimmed videos of well-defined and at-times predictable sequences. For example the action 'wash aubergine' can be part of the following sequence -you first 'take the aubergine', 'turn on the tap', 'wash the aubergine' and finally 'turn off the tap' (Fig. 1). Furthermore, the objects (the aubergine and tap in this case) are persistent over some of the neighbouring actions.

OWL (Observe, Watch, Listen): Localizing Actions in Egocentric Video via Audiovisual Temporal Context

“…In egocentric videos, temporal context might be even more informative. For instance, at being unedited and continuous, actions unfold, with a more often than not, predictable sequence [20,21,24]. Beyond visual cues, we argue that context from the audio stream also provide priors to better localize actions.…”

“…Deep learning facilitates audiovisual learning as it enables learning per-modality hierarchical representations [37], which are more optimal than designing hand-crafted features. Recent works provide us with more sophisticated solutions where the learned modality representations are being fused implicitly by the network and are optimized for the downstream task, such as [1,17,24,25,32,42,44]. While several works discussed the audiovisual scenario for the action recognition task [44], incorporating audio for TAL is not a widely researched area.…”

“…Furthermore, using temporal context has been proven to be effective for both action recognition, and localization [12,13,24,34,43,[46][47][48]. In egocentric videos, temporal context might be even more informative.…”

Bringing Online Egocentric Action Recognition into the wild

“…The complex nature of egocentric videos raises a variety of challenges, such as ego-motion [18], partially visible or occluded objects, and environmental bias [19], [20], [21], [22], which limit the performance of traditional approaches when used in FPAR [23], [24]. Those challenges attract the community's interest and motivate the design of novel and more complex architectures, often based on multi-stream approaches such as [25], [15], [26], [27], [16].…”

E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

“…The complex nature of egocentric videos raises a variety of challenges, such as egomotion [60], partially visible or occluded objects, and environmental bias [53,72,77,84,88], which limit the performance of traditional, third-person, approaches when used in first person action recognition (FPAR) [14,15]. The community's interest has quickly grown [16,17,19,83] in recent years, thanks to the possibilities that these data open for the evaluation and understanding of human behavior, leading to the design of novel architectures [30,51,52,91,104]. While the use of optical flow has been the de-facto procedure [14][15][16][17]19,41] in FPAR, the interest has recently shifted towards more lightweight alternatives, such as gaze [27,59,71], audio [9,52,77], depth [32], skeleton [32], and inertial measurements [41], to enable motion modeling in online settings.…”