Predictive-Corrective Networks for Action Detection

Abstract

While deep feature learning has revolutionized techniques for static-image understanding, the same does not quite hold for video processing. Architectures and optimization techniques used for video are largely based off those for static images, potentially underutilizing rich video information. In this work, we rethink both the underlying network architecture and the stochastic learning paradigm for temporal data. To do so, we draw inspiration from classic theory on linear dynamic systems for modeling time series. By extending such models to include nonlinear mappings, we derive a series of novel recurrent neural networks that sequentially make top-down predictions about the future and then correct those predictions with bottom-up observations. Predictive-corrective networks have a number of desirable properties: (1) they can adaptively focus computation on "surprising" frames where predictions require large corrections, (2) they simplify learning in that only "residual-like" corrective terms need to be learned over time and (3) they naturally decorrelate an input data stream in a hierarchical fashion, producing a more reliable signal for learning at each layer of a network. We provide an extensive analysis of our lightweight and interpretable framework, and demonstrate that our model is competitive with the two-stream network on three challenging datasets without the need for computationally expensive optical flow.Comment: Accepted to CVPR 2017. [v2]: Updated Multi-LSTM mAP on MultiTHUMOS (should be 29.7, was initially reported as 29.6). [Project URL]: http://www.achaldave.com/projects/predictive-corrective

Temporal Recurrent Networks for Online Action Detection

“…Single-frame CNN [38] 34.7 Two-stream CNN [37] 36.2 C3D + LinearInterp [34] 37.0 Predictive-corrective [10] 38.9 LSTM [14] 39. [32] are much larger when the input contains sensor data.…”

Rethinking the Faster R-CNN Architecture for Temporal Action Localization

“…Another line of work generates frame-wise or snippetwise action labels, and uses these labels to define the temporal boundaries of actions [29,38,10,26,55,20]. One major challenge here is to enable temporal contextual reasoning in predicting the individual labels.…”

“…One major challenge here is to enable temporal contextual reasoning in predicting the individual labels. Lea et al [26] proposed novel temporal convolutional architectures to capture longrange temporal dependencies, while others [29,38,10] use recurrent neural networks. A few other methods add a separate contextual reasoning stage on top of the frame-wise or snippet-wise prediction scores to explicitly model action durations or temporal transitions [34,55,20].…”

TAN: Temporal Aggregation Network for Dense Multi-Label Action Recognition

“…Sigurdsson et al [27] used a fully-connected temporal CRF model for reasoning over various aspects of activities. Dave et al [10] employed RNNs to sequentially make top-down predictions and later then corrected them by bottom-up observations. Most recently, Sigurdsson et al [28] performed a detailed analysis on what kinds of information are needed to achieve substantial gains for activity understanding among objects, verbs, intent, and sequential reasoning.…”

“…Comparison with state-of-the-art. On Charades, we compare with several state-of-the-art methods such as [29,35,38,13,27,15,10]. Our results are shown in Table 6 Table 7.…”