Encouraging LSTMs to Anticipate Actions Very Early

Abstract

In contrast to the widely studied problem of recognizing an action given a complete sequence, action anticipation aims to identify the action from only partially available videos. As such, it is therefore key to the success of computer vision applications requiring to react as early as possible, such as autonomous navigation. In this paper, we propose a new action anticipation method that achieves high prediction accuracy even in the presence of a very small percentage of a video sequence. To this end, we develop a multi-stage LSTM architecture that leverages context-aware and action-aware features, and introduce a novel loss function that encourages the model to predict the correct class as early as possible. Our experiments on standard benchmark datasets evidence the benefits of our approach; We outperform the state-of-the-art action anticipation methods for early prediction by a relative increase in accuracy of 22.0% on JHMDB-21, 14.0% on UT-Interaction and 49.9% on UCF-101.

Predicting the Future: A Jointly Learnt Model for Action Anticipation

“…Earliest Latest Context-aware + loss in [21] 30.6 71.1 Context-aware + loss in [34] 22.6 73.1 Multi stage LSTM [3] 80.5 83.4 Proposed 84.2 85.6 Table 1. Action anticipation results for UCF101 considering the 'Earliest' 20% of frames and 'Latest' 50% of frames.…”

“…Lee et al [30] proposed a human activity representation method, termed sub-volume co-occurrence matrix, and developed a method to predict partially observed actions with the aid of a pretrained CNN. The deep network approach of Aliakbarian et al [3] used a multi-stage LSTM architecture that incorporates context-aware and action-aware features to predict classes as early as possible. The CNN based action anticipation model of [40] predicts the most plausible future motion, and was improved via an effective loss function based on dynamic and classification losses.…”

“…The dataset has total of 120 videos. For the state-of-the-art comparison we utilise a 10-fold leave-one-out cross validation on each set and the mean performance over all sets is obtained, as per [3].…”

“…To evaluate our model on each dataset, where possible we consider two settings for the number of input frames, namely the 'Earliest' and 'Latest' settings. For UCF101 and UTI, similar to [3] we consider 20% and 50% of the frames for the 'Earliest' and 'Latest' settings, respectively; following [3] we do not use more than 50 frames for the 'Latest' setting. For each dataset and setting, we resample the input videos such that all sequences have a constant number of frames.…”

“…1(a)), the aim of action anticipation ( Fig. 1(b)) is to predict the future action as early as possible by observing only a portion of the action [3]. Therefore, for the prediction we only have partial information in the form of a small number of frames, so the available information is scarce.…”