Deep Neural Networks Using Capsule Networks and Skeleton-Based Attentions for Action Recognition

Abstract

This work develops Deep Neural Networks (DNNs) by adopting Capsule Networks (Cap-sNets) and spatiotemporal skeleton-based attention to effectively recognize subject actions from abundant spatial and temporal contexts of videos. The proposed generic DNN includes four 3D Convolutional Neural Networks (3D_CNNs), Attention-Jointed Appearance (AJA) and Attention-Jointed Motion (AJM) generation layers, two Reduction Layers (RLs), two Attention-based Recurrent Neural Networks (A_RNNs), and an inference classifier, where RGB, transformed skeleton, and optical-flow channel streams are inputs. The AJA and AJM generation layers emphasize skeletons to the appearances and motions of a subject, respectively. A_RNNs generate attention weights over time steps to highlight rich temporal contexts. To integrate CapsNets in this generic DNN, three types of CapsNet-based DNNs are devised, where the CapsNets take over a classifier, A_RNN+classifier, and RL+A_RNN+classifier. The experimental results reveal that the proposed DNN using CapsNet as an inference classifier outperforms the other two CapsNetbased DNNs and the generic DNN adopting the feedforward neural network as an inference classifier. Additionally, our best CapsNet-based DNN achieves average accuracies of 98.5% for the state-of-theart performance in UCF101, 82.1% for near-state-of-the-art performance in HMDB51, and 95.3% for panoramic videos, to the best of our knowledge. Particularly, it is determined that the generic CapsNet behaves as an outstanding inference classifier but is slightly worse than the A_RNN in interpreting temporal evidence for recognition. Therefore, the proposed DNN, which employs CapsNet to fulfill an inference classifier, can be superiorly applied to various context-aware visual applications.INDEX TERMS Deep neural network, convolutional neural network, recurrent neural network, capsule network, spatiotemporal attention, skeleton, action recognition.

[Retracted] Detection Method of Limb Movement in Competitive Sports Training Based on Deep Learning

“…Sports Training. In the process of human motion capture, because the inertial sensor is self-sufficient and has no external reference point, it cannot obtain spatial displacement information, so it is necessary to use positioning technology to obtain displacement information in the process of human motion capture [20]. Wide-area positioning and short-range positioning are two types of wireless positioning technologies.…”

Attention correlated appearance and motion feature followed temporal learning for activity recognition

“…In order to improve performance in visual perception, several generations of CNNs have been created with the input vectors taking care of one image or multiple images. Particularly, multiple images are commonly adopted as an input vector which has the embedded temporal information as well as the spatial information [2], [4]. In addition to improving learning, many researchers used temporal networks to perform large-scale visual learning and activity classification from video clips, where temporal networks had recurrent connections to aid in video context understanding regarding time [2], [4]- [7].…”

“…The motion being performed can be at a fast-refreshing speed, and individual frames can be ambiguous. Therefore, motion cues provide a necessary approach by allowing the compensated optical flows to pick up potential [2], [4]. Another important reason is that current CNNs architectures are not able to take full advantage of temporal information and their performance is consequently often dominated by appearance recognition.…”

“…Instead of the usual feed-forward neural network with dropout and softmax making the final prediction, in the inference classifier, the predicting voting classifier (PVC) scheme based on the multiple nearest class mean (NCM) classifiers [8], SoftMax, and majority voting [9] are developed to determine the action class. In this study, two datasets from HMDB51 and UCF101 [2] are adopted to evaluate the proposed deep neural network (DNN). To ensure a fair comparison, the UCF-101 dataset is used to validate the temporal network structures of 12 different types, which are then simulated and compared to the best one.…”

Prediction of Methylene Blue Removal by Nano TiO2 Using Deep Neural Network

“…On the other hand, DNN models have achieved human-level performance and have shown great success in different real-world applications, including computer vision [ 16 ], textile process, biomedical engineering [ 17 ], material engineering [ 18 ]. DNN is an efficient machine learning tool suitable for the prediction of output parameters from input variables where there is an unknown relationship exists between input and output variables [ 19 , 20 , 21 ]. In recent years, DNN has been widely used to predict various properties of textiles.…”