SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos

Unified Recurrence Modeling for Video Action Anticipation

“…In this case the scaled dot-product in the self-attention computes the pairwise similarity of all vertices from the inputs, which can be viewed as an implicit edge estimation. This can be extended by optionally providing the edge estimation explicitly, in the form of an adjacency matrix A = {a vw ; ∀w ∈ N (v), v ∈ G}, and using it during the attention computation as described in (11) and (12). We refer to these two cases as implicit and explicit edge learning.…”

“…RU-LSTM [19] deploys two LSTMs and behaves as an encoder-decoder, where the first progressively summarizes the observed together with the second that unrolls over future predictions without observing. The unrolling design can also be found in [11], [20], but with the rolling part replaced by SlowFast [21] and Higher-Order Recurrent Transformer, respectively. [22] aggregates the multiple predictions by pooling over different granularity of temporal segments to improve the anticipation accuracy.…”

“…The output from the readout function is then mean-pooled and fed to an unrolling classifier to obtain verb, noun, and action predictions. The unrolling classifier is a widely adopted design in action anticipation [19], [11], [20], and we follow the similar classifier design as prior works. The classifier outputs are supervised with ground-truth labels using cross-entropy.…”

“…Recurrent neural networks are widely adopted for modeling the temporal relationship in anticipation problems [3], [10], [11] and lead to better performance than clip-based methods [12], [13], as opposed to the mainstream in action recognition. The basic assumption is that in action anticipation the observations are incomplete and indirect, although related, to the target action.…”