Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Abstract

The paucity of videos in current action classification datasets (UCF-101 and HMDB-51) has made it difficult to identify good video architectures, as most methods obtain similar performance on existing small-scale benchmarks. This paper re-evaluates state-of-the-art architectures in light of the new Kinetics Human Action Video dataset. Kinetics has two orders of magnitude more data, with 400 human action classes and over 400 clips per class, and is collected from realistic, challenging YouTube videos. We provide an analysis on how current architectures fare on the task of action classification on this dataset and how much performance improves on the smaller benchmark datasets after pre-training on Kinetics.We also introduce a new Two-Stream Inflated 3D ConvNet (I3D) that is based on 2D ConvNet inflation: filters and pooling kernels of very deep image classification ConvNets are expanded into 3D, making it possible to learn seamless spatio-temporal feature extractors from video while leveraging successful ImageNet architecture designs and even their parameters. We show that, after pre-training on Kinetics, I3D models considerably improve upon the state-of-the-art in action classification, reaching 80.9% on HMDB-51 and 98.0% on UCF-101.

Action Recognition with Dynamic Image Networks

“…We outperform the state-of-the-art methods with a significant margin with the exception of I3D+ [4] (98% and 80.7%). Note that this method is pre-trained on additional 300,000 videos and relies on a two-stream variant.…”

“…Furthermore, our four stream models do not improve significantly after the inclusion of improved trajectories (95.5% → 96.0% and 72.5% → 74.9%), showing that the vast majority of the benefit is intrinsic to the proposed architecture. This is interesting, as our four stream models are one of the first models together with I3D [4] which manages to surpass the 95% and 70% barriers on respective UCF101 and HMDB51 without relying on handcrafted features.…”

“…In addition to ResNeXt-50 model, here we also train our model with the deeper ResNeXt-101 [75] and report its performance as well. In order to provide a fair comparison, we split the table into two parts, the ones incorporate their methods Method UCF101 HMDB51 CNN-hid6 [80] 79.3 -Comp-LSTM [62] 84.3 44.0 C3D+SVM [65] 85.2 -2S-CNN [78] 88.0 59.4 FSTCN [63] 88.1 59.1 2S-CNN+Pool [78] 88.2 -Objects+Motion(R * ) [26] 88.5 61.4 2S-CNN+LSTM [78] 88.6 -TDD [70] 90 [48] 86.0 60.1 FM+IDT [47] 87.9 61.1 MIFS+IDT [35] 89.1 65.1 CNN-hid6+IDT [80] 89.6 -C3D Ensemble+IDT (Sports-1M) [65] 90.1 -C3D+IDT+SVM [65] 90.4 -TDD+IDT [70] 91.5 65.9 Sympathy [9] 92.5 70.4 Two-Stream Fusion+IDT [15] 93.5 69.2 ST-ResNet+IDT [14] 94 [4] has been pre-trained on a large-scale video dataset, Kinetics300k.…”

Human action recognition using fusion of modern deep convolutional and recurrent neural networks

“…It also outperforms early 3D CNN architecture [9]. [10] and [11] obtain better results, however, these are much more complicated models which were pre-trained on much bigger dataset (Kinetics); consequently, their results are not directly comparable. The proposed model achieves good results without pre-training on bigger datasets, and thus it is better suited for use cases where the limited amount of training data is available.…”

“…C3D [9] uses relatively shallow CNN architecture, trained from scratch on large video datasets, that is applied to nonoverlapping frame clips with the classification result computed by averaging the scores predicted for all clips. I3D [10] proposes inflating 2D CNNs into 3D and bootstrapping 3D filters from 2D filters, which provides parameter initialization from 2D models trained on ImageNet. [11] explores even deeper ResNet-based architectures similar to those that worked well for image recognition.…”