Semantic Image Networks for Human Action Recognition

Abstract

In this paper, we propose the use of a semantic image, an improved representation for video analysis, principally in combination with Inception networks. The semantic image is obtained by applying localized sparse segmentation using global clustering (LSSGC) prior to the approximate rank pooling which summarizes the motion characteristics in single or multiple images. It incorporates the background information by overlaying a static background from the window onto the subsequent segmented frames. The idea is to improve the action-motion dynamics by focusing on the region which is important for action recognition and encoding the temporal variances using the frame ranking method. We also propose the sequential combination of Inception-ResNetv2 and long-short-term memory network (LSTM) to leverage the temporal variances for improved recognition performance. Extensive analysis has been carried out on UCF101 and HMDB51 datasets which are widely used in action recognition studies. We show that (i) the semantic image generates better activations and converges faster than its original variant, (ii) using segmentation prior to approximate rank pooling yields better recognition performance, (iii) The use of LSTM leverages the temporal variance information from approximate rank pooling to model the action behavior better than the base network, (iv) the proposed representations can be adaptive as they can be used with existing methods such as temporal segment networks to improve the recognition performance, and (v) our proposed four-stream network architecture comprising of semantic images and semantic optical flows achieves state-of-the-art performance, 95.9% and 73.5% recognition accuracy on UCF101 and HMDB51, respectively. We propose a method (LSSGC) for dynamically segmenting the image.  We introduce a semantic image to represent motion dynamics over a sequence of frames.  We propose an SR-LSTM based semantic image network to leverage temporal information and variance.  We use the semantic images with four-stream and temporal segment networks (a state-of-the-art network design) to show that the proposed representation is adaptable.  We report state-of-the-art results using semantic image networks.

Metric-Based Attention Feature Learning for Video Action Recognition

“…The main difference between the existing studies [46,47] and the proposed method lies in the location of separating the object and the background. Specifically, [46,47] synthesizes the features of discriminative action at the network input stage, while the proposed method adaptively learns discerning features within the network. Section 4.C experimentally analyzes this phenomenon.…”

“…Several techniques have been proposed to distinguish between objects and backgrounds by appropriately manipulating input information [46,47]. For example, the dynamic image [46], which is an image created by synthesizing appearance and dynamics, i.e., static and temporal information, can express the temporal information of a single RGB image, and multimodal input based on this can contribute to improve the performance of the action recognition task.…”

“…For example, the dynamic image [46], which is an image created by synthesizing appearance and dynamics, i.e., static and temporal information, can express the temporal information of a single RGB image, and multimodal input based on this can contribute to improve the performance of the action recognition task. In addition, the semantic image [47] generated by applying a specific localized sparse segmentation technique to an RGB image was effectively used to extract only significant action changes by separating the behavioral regions and background. The main difference between the existing studies [46,47] and the proposed method lies in the location of separating the object and the background.…”

“…In addition, the semantic image [47] generated by applying a specific localized sparse segmentation technique to an RGB image was effectively used to extract only significant action changes by separating the behavioral regions and background. The main difference between the existing studies [46,47] and the proposed method lies in the location of separating the object and the background. Specifically, [46,47] synthesizes the features of discriminative action at the network input stage, while the proposed method adaptively learns discerning features within the network.…”

LaM-2SRN: A Method Which Can Enhance Local Features and Detect Moving Objects for Action Recognition

“…Bilen, H et al [7] use the pre-trained 2D CNN to recognize human actions, they introduce the concept of dynamic image, which is obtained by Rank Pooling [8] on video frames. Sunder Ali Khowaja and Seok-Lyong Lee [9] propose the semantic image for video analysis, which is obtained by applying localized sparse segmentation using global clustering prior to the approximate rank pooling. Recently, more and more 3D CNN are applied in the action recognition, like C3D [10] (Convolutional 3D), I3D (Inflated-3D) [11], ResNet [12] and so forth, where ResNet can be divided into 2D ResNet and 3D ResNet, and all of them have a common concept, that is Residual Learning.…”