LSTA: Long Short-Term Attention for Egocentric Action Recognition

Abstract

Egocentric activity recognition is one of the most challenging tasks in video analysis. It requires a fine-grained discrimination of small objects and their manipulation. While some methods base on strong supervision and attention mechanisms, they are either annotation consuming or do not take spatio-temporal patterns into account. In this paper we propose LSTA as a mechanism to focus on features from relevant spatial parts while attention is being tracked smoothly across the video sequence. We demonstrate the effectiveness of LSTA on egocentric activity recognition with an end-to-end trainable two-stream architecture, achieving state-of-the-art performance on four standard benchmarks.

Seeing and Hearing Egocentric Actions: How Much Can We Learn?

“…These results indicate that directly training over the action performs better than the combination of verb+noun. Our multimodal models obtained better scores than the challenge baseline and has similar results as previous works [50]. Additionally, the results obtained on the unseen participants (S2) test split are in the top-ten ranking of the first challenge.…”

Multitask Learning to Improve Egocentric Action Recognition

“…In [57], the relationship between tasks is modelled in a latent space to transfer knowledge between them and reduce the number of required training samples. MTL in egocentric vision appears in [1,28,25,18,29,47].…”

“…Recurrent with attention The temporal aspect of videos is further studied with recurrent attention mechanisms [3,39,48,47,25,11,56] that act to find the most informative parts in images (spatial attention) or the most informative frames throughout videos (temporal attention). An encoderdecoder scheme is described in [3] for textual description of videos.…”

“…The combined focus on these regions allows the network to discard redundant frames of the input video segment that would otherwise obfuscate the prediction task. Spatial attention is considered in [48] where the important regions from every frame are given as input to an LSTM for action recognition, whereas in [47] spatial attention is further correlated sequentially through continuous frames. In [56], attention is based on video specific spatial features (such as person detections in third-person videos and objects with motion for egocentric), which are calculated intermittently over the course of a video.…”

What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention

“…Action Recognition Our work is related to previous research on action recognition from third person vision [7,13,14,30,34,35,49,57,59,60,61,67] and first person vision [11,12,36,37,38,44,48,52,54,55,56]. Specifically, we build on previous ideas investigated in the context of action recognition such as the use of multiple modali-ties for video analysis [49], the use of Temporal Segment Networks [61] as a principled way to train CNNs for action recognition, as well as the explicit encoding of object-based features [11,38,44,51,56] to analyze egocentric video.…”