DynamoNet: Dynamic Action and Motion Network

Abstract

In this paper, we are interested in self-supervised learning the motion cues in videos using dynamic motion filters for a better motion representation to finally boost human action recognition in particular. Thus far, the vision community has focused on spatio-temporal approaches using standard filters, rather we here propose dynamic filters that adaptively learn the video-specific internal motion representation by predicting the short-term future frames. We name this new motion representation, as dynamic motion representation (DMR) and is embedded inside of 3D convolutional network as a new layer, which captures the visual appearance and motion dynamics throughout entire video clip via end-to-end network learning. Simultaneously, we utilize these motion representation to enrich video classification. We have designed the frame prediction task as an auxiliary task to empower the classification problem.With these overall objectives, to this end, we introduce a novel unified spatio-temporal 3D-CNN architecture (Dy-namoNet) that jointly optimizes the video classification and learning motion representation by predicting future frames as a multi-task learning problem. We conduct experiments on challenging human action datasets: Kinetics 400, UCF101, HMDB51. The experiments using the proposed DynamoNet show promising results on all the datasets.

SAST: Learning Semantic Action-Aware Spatial-Temporal Features for Efficient Action Recognition

“…The superior results in Table 4 show that we achieve the comparable performances: 75.7% on the top-1 accuracy and 93.8% on the top-5 accuracy in validation set, which outperforms our baseline ECO-Lite-EN [17] by 5.7% on top-1 accuracy and by 4.4% on top-5 accuracy. It also outperforms the recent works STM [31] and DynamoNet-32F (ResNext101) [29] by 2.0% and 7.5% on top-1 accuracy and by 2.2% and 5.7% on top-5 accuracy, respectively.…”

“…Specifically, our SAST-EN significantly outperforms our baseline ECO-Lite-EN [17] by 1.6% on UCF101 and by 2.7% on HMDB51. It also outperforms the recent works STM [31], DistInit [28] and DynamoNet-32F (ResNext101) [29] by 0.2%, 10.6% and 3.3% on UCF101, and by 2.9%, 20.3% and 6.6% on HMDB51, respectively. Note that SAST-EN represents the average scores obtained from an ensemble of SAST network with the {16, 20, 24, 32} number of input frames similar to ECO-Lite-EN [17].…”

“…Girdhar et al [28] propose an approach DistInit to transfer image models to video, which takes the years of effort in collecting and labeling large and clean still-image datasets. Reference [29] presents a dynamic action and motion network (DynamoNet) using } over time and randomly select one frame from each segment and feed total of N frames into the weight shared 2D Deformable Convolutional network (2DDC) to extract semantic action-aware spatial features. Secondly, these discriminative spatial features are concatenated with time series to yield a preliminary video-level representation.…”

Local-enhanced Interaction for Temporal Moment Localization

“…Condconv [30] improves the model capacity by increasing the size and complexity of the kernel-generating function. Due to their advantages, dynamic filter networks have been applied in many areas, like human action recognition [7], super-resolution [29].…”

Online supervised attention-based recurrent depth estimation from monocular video

“…Recently, there were few works, which focused on exploiting the temporal information via concatenation of multiple frames at input, such as Sun et al (2015) and Diba et al (2019). The problem of these approaches lies in inability to scale well on long sequences.…”