Action Recognition in Videos with Spatio-Temporal Fusion 3D Convolutional Neural Networks

Wang, Y.; Shen, Xuanjing; Chen, H. P.; Sun, Jiaxi

doi:10.1134/s105466182103024x

Cited by 5 publications

(4 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the video, the 2D skeleton information is first extracted, then the 2D skeleton information is converted into 3D skeleton information, and then the 3D information is used as the input of the algorithm model for action recognition. The 2D attitude estimation algorithm uses the RMPE algorithm, which belongs to the top-down attitude estimation method and is an improvement of the SPPE algorithm to solve the problem of inaccurate and redundant detection frame positions [21]. It is divided into three steps: human frame detection, human pose estimation, and non-maximum suppression (Figure 3).…”

Section: Methodsmentioning

confidence: 99%

Three-Dimensional Action Recognition for Basketball Teaching Coupled with Deep Neural Network

Zuo

2022

Electronics

View full text Add to dashboard Cite

This study proposes a 3D attitude estimation algorithm using the RMPE algorithm coupled with a deep neural network that combines human pose estimation and action recognition, which provides a new idea for basketball auxiliary training. Compared with the traditional single-action recognition method, the present method makes the recognition accuracy better and the display effect more intuitive. The flipped classroom teaching mode based on this algorithm is applied to the college sports basketball optional course to explore the influence of this teaching mode on the classroom teaching effect. Compared with the evaluation index of action recognition, the experimental results of various action recognition methods and datasets are compared and analyzed, and it is verified that the method has a good recognition effect. The values of Topi and Top5 of the proposed method are 42.21% and 88.77%, respectively, which are 10.61% and 35.09% higher than those of the Kinetics-skeleton dataset. However, compared with the NTU RGM dataset, the recognition rate of Topi is significantly reduced. Compared with the traditional single-action recognition method, this method has better recognition accuracy and a more intuitive display effect. The fusion method of human posture estimation and motion recognition provides a new idea for basketball auxiliary training.

show abstract

Section: Methodsmentioning

confidence: 99%

Three-Dimensional Action Recognition for Basketball Teaching Coupled with Deep Neural Network

Zuo

2022

Electronics

View full text Add to dashboard Cite

show abstract

“…In contrast to 2D CNN, the convolutional filters of 3D CNN are 3D. The third dimension in the convolutional filters of 3D CNN is the time depth [44,45]. This allows for constant temporal characteristics, enabling 3D CNN to extract spatiotemporal coupling relationships from input data [46].…”

Section: The Delay Prediction Modelmentioning

confidence: 99%

A Novel Hybrid Deep Learning Model for Complex Systems: A Case of Train Delay Prediction

Wang,

Guo,

Zhang

2024

Advances in Civil Engineering

View full text Add to dashboard Cite

Predicting the status of train delays, a complex and dynamic problem, is crucial for railway enterprises and passengers. This paper proposes a novel hybrid deep learning model composed of convolutional neural networks (CNN) and temporal convolutional networks (TCN), named the CNN + TCN model, for predicting train delays in railway systems. First, we construct 3D data containing the spatiotemporal characteristics of real-world train data. Then, the CNN + TCN model employs a 3D CNN component, which is fed into the constructed 3D data to mine the spatiotemporal characteristics, and a TCN component that captures the temporal characteristics in railway operation data. Furthermore, the characteristic variables corresponding to the two components are selected. Finally, the model is evaluated by leveraging data from two railway lines in the United Kingdom. Numerical results show that the CNN + TCN model has greater accuracy and convergence performance in train delay prediction.

show abstract

“…Zebhi et al [23] used Gait History Image (GHI) and gradient to extract spatiotemporal features and the time-sliced averaged gradient boundary to characterise motion, using a VGG-16 classifier. Wang et al [24] extracted spatial features using gradient and extracted motion features using optical flow, which were then classified using two different 3D CNNs. Khan et al [25] used DenseNet201 and InceptionV3 to extract features and a Kurtosis-controlled Weighted KNN as the final classifier.…”

Section: State Of the Artmentioning

confidence: 99%

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Alavigharahbagh,

Hajihashemi,

Machado

et al. 2023

Information

View full text Add to dashboard Cite

In this article, a hierarchical method for action recognition based on temporal and spatial features is proposed. In current HAR methods, camera movement, sensor movement, sudden scene changes, and scene movement can increase motion feature errors and decrease accuracy. Another important aspect to take into account in a HAR method is the required computational cost. The proposed method provides a preprocessing step to address these challenges. As a preprocessing step, the method uses optical flow to detect camera movements and shots in input video image sequences. In the temporal processing block, the optical flow technique is combined with the absolute value of frame differences to obtain a time saliency map. The detection of shots, cancellation of camera movement, and the building of a time saliency map minimise movement detection errors. The time saliency map is then passed to the spatial processing block to segment the moving persons and/or objects in the scene. Because the search region for spatial processing is limited based on the temporal processing results, the computations in the spatial domain are drastically reduced. In the spatial processing block, the scene foreground is extracted in three steps: silhouette extraction, active contour segmentation, and colour segmentation. Key points are selected at the borders of the segmented foreground. The last used features are the intensity and angle of the optical flow of detected key points. Using key point features for action detection reduces the computational cost of the classification step and the required training time. Finally, the features are submitted to a Recurrent Neural Network (RNN) to recognise the involved action. The proposed method was tested using four well-known action datasets: KTH, Weizmann, HMDB51, and UCF101 datasets and its efficiency was evaluated. Since the proposed approach segments salient objects based on motion, edges, and colour features, it can be added as a preprocessing step to most current HAR systems to improve performance.

show abstract

Action Recognition in Videos with Spatio-Temporal Fusion 3D Convolutional Neural Networks

Cited by 5 publications

References 26 publications

Three-Dimensional Action Recognition for Basketball Teaching Coupled with Deep Neural Network

Three-Dimensional Action Recognition for Basketball Teaching Coupled with Deep Neural Network

A Novel Hybrid Deep Learning Model for Complex Systems: A Case of Train Delay Prediction

Deep Learning Approach for Human Action Recognition Using a Time Saliency Map Based on Motion Features Considering Camera Movement and Shot in Video Image Sequences

Contact Info

Product

Resources

About