Action Recognition Based on Efficient Deep Feature Learning in the Spatio-Temporal Domain

Abstract

Abstract-Hand-crafted feature functions are usually designed based on the domain knowledge of a presumably controlled environment and often fail to generalize, as the statistics of realworld data cannot always be modeled correctly. Data-driven feature learning methods, on the other hand, have emerged as an alternative that often generalize better in uncontrolled environments. We present a simple, yet robust, 2D convolutional neural network extended to a concatenated 3D network that learns to extract features from the spatio-temporal domain of raw video data. The resulting network model is used for contentbased recognition of videos. Relying on a 2D convolutional neural network allows us to exploit a pretrained network as a descriptor that yielded the best results on the largest and challenging ILSVRC-2014 dataset. Experimental results on commonly used benchmarking video datasets demonstrate that our results are state-of-the-art in terms of accuracy and computational time without requiring any preprocessing (e.g., optic flow) or a priori knowledge on data capture (e.g., camera motion estimation), which makes it more general and flexible than other approaches. Our implementation is made available.

Scene Understanding Using Deep Learning

“…We also see this transfer learning technique being applied successfully for recognizing actions. This is achieved by using a pretrained image recognition model for the individual frames of videos [7,28,29].…”

“…Attempts have been made to make action recognition invariant to different kinds of situations. This includes the usage of optical flow as additional information [28] or using 3D (spatio-temporal) convolutional kernels [7,30]. Recurrent Neural Networks have also been explored to learn from long term dependencies in different types actions [31].…”

“…Accuracy CNN with transfer learning [29] 65.4% LRCN (RGB) [36] 71.1% Spatial stream ConvNet [28] 72.6% LSTM composite model [37] 75.8% C3D (1 net) [30] 82.3% Temporal stream ConvNet [28] 83.7% C3D (3 nets) [30] 85.2% Combined ordered and improved trajectories [38] 85.4% Stacking classifiers and CRF smoothing [39] 85.7% Improved dense trajectories [40] 85.9% Improved dense trajectories with human detection [41] 86.0% 2D followed by 3D convolutions [7] 86.7% Spatial and temporal stream fusion [28] 88.0%…”

“…This is because they are low-cost and capture data in the form of dense depth maps, in addition to color images. Convolutional Neural Networks (CNNs) provide a robust way to extract useful information from the data acquired using these devices [4,5,6,7]. In this chapter we will discuss the basic idea behind standard feed forward CNNs (Section 2) and their application in semantic segmentation (Section 3) and action recognition (Section 4).…”

“…In this chapter we will discuss the basic idea behind standard feed forward CNNs (Section 2) and their application in semantic segmentation (Section 3) and action recognition (Section 4). Further in depth analysis and stateof-the-art solutions for these applications can be found in our recent publications [6] and [7].…”