Learning Activity Progression in LSTMs for Activity Detection and Early Detection

Abstract

In this work we improve training of temporal deep models to better learn activity progression for activity detection and early detection tasks. Conventionally, when training a Recurrent Neural Network, specifically a Long Short Term Memory (LSTM) model, the training loss only considers classification error. However, we argue that the detection score of the correct activity category, or the detection score margin between the correct and incorrect categories, should be monotonically non-decreasing as the model observes more of the activity. We design novel ranking losses that directly penalize the model on violation of such monotonicities, which are used together with classification loss in training of LSTM models. Evaluation on ActivityNet shows significant benefits of the proposed ranking losses in both activity detection and early detection tasks.

Early Action Prediction by Soft Regression

“…This implies that our proposed soft regression framework is also beneficial for the task of action recognition and can obtain the state-of-the-art recognition result. As expected, our soft regression model outperformed the RankLSTM [36] and DeepSCN [25] approaches again, which demonstrates the efficacy of our soft label learning framework for early action prediction. We also note that the prediction results of most methods on the first 10% frames on this set is much lower than that on the ORGBD and SYSU 3D HOI sets.…”

“…and RankLSTM [36] with the same inputs, which demonstrates the effectiveness of our model for predicting early actions.…”

“…We also compared our soft regression model with the DeepSCN [25] and RankLSTM [36] models. The source codes for DeepSCN and RankLSTM are not available for benchmarking, we re-implemented them strictly by following the descriptions in [25], [36]. For a fair comparison, we fed our LAFF features into their learning frameworks and reported the best results among a large range parameter settings.…”

“…However, they assumed that the progress level of ongoing action is provided along with the observed sequence in the testing phase, which renders their method less applicable in the real-world applications, since it is hard to access the progress level of ongoing action until it has been fully executed and observed when applied on unknown sequence. Recently, some researchers intend to conduct early action prediction [22] or detection [36] by learning action progress levels in an automatic manner. Yet, their prediction/detection is easily hindered by the performance of learning action progression, which is still a challenging problem needed to be carefully addressed.…”

PMHI: Proposals From Motion History Images for Temporal Segmentation of Long Uncut Videos

“…The exhaustive computation of video classifiers (sliding window) has been avoided in [11][12][13][14][15][16]. In those methods, first a number of candidate segments containing human actions, known as action proposals, are produced.…”