End-to-End Multi-Resolution 3D Capsule Network for People Action Detection

Abstract

In this paper, we propose an end-to-end multi-resolution three-dimensional (3D) capsule network for detecting actions of multiple actors in a video scene. Unlike previous capsule, network-based action recognition does not specifically concern with the individual action of multiple actors in a single scene, our 3D capsule network takes advantage of multi-resolution technique to detect different actions of multiple actors that have different sizes, scales, and aspect ratios. Our 3D capsule network is built on top of 3D convolutional neural network (3DCNN) that extracts spatio-temporal features from video frames inside regions of interest generated by Faster RCNN object detection. We first apply our method to the problem of detecting illegal cheating activities in a classroom examination scene with multiple subjects involved. Second, we test our system on the publicly available and extensively studied UCF-101 dataset. We compare our method with several state-of-the-art 3DCNN-based methods, first the multi-resolution 3DCNN, the single-resolution 3D capsule network, and a combination of both these models. We show that models containing 3D capsule networks have a slight advantage over the conventional 3DCNN and multi-resolution 3DCNN. Our 3D capsule networks not only perform a classification of said actions but also generate videos of single actions. Our experimental results show that the use of multi-resolution pathways in the 3D capsule networks make the result even better. Such findings also hold even when we use pre-trained C3D (convolutional 3D) features to train these networks. We believe that the multiple resolutions capture lower-level features at different scales. At the same time, the 3D capsule layers combine these features in more complex ways than conventional convolutional models.

Convolutional neural network-based recognition method for volleyball movements

“…In behavior recognition, the joint probability of action sequence is calculated to recognize actions. In the following research, many improvements have been made to the Hidden Markov Model [ 13 ]. There are many improved models based on Markov Model, but the Hidden Markov Model can deal with two independent processes at most.…”