Action Recognition Using Form and Motion Modalities

Meng, Qing-Hao; Zhu, Heyan; Zhang, Weigang; Piao, Xuefeng; Zhang, Aijie

doi:10.1145/3350840

Cited by 14 publications

(5 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The classifier demonstrates above-human performance and surpasses the sparse spatio-temporal feature approach detailed in Burgos-Artizzu et al 6 on the CRIM13 dataset. It also performs better than prior methods based on temporal features 25 , independent component analysis 26 , hierarchical sparse coding 27 , integrated sparse and dense trajectory features 28 .…”

Section: Resultsmentioning

confidence: 96%

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

Harris

Finn

Kieseler

et al. 2023

Sci Rep

View full text Add to dashboard Cite

The identification of animal behavior in video is a critical but time-consuming task in many areas of research. Here, we introduce DeepAction, a deep learning-based toolbox for automatically annotating animal behavior in video. Our approach uses features extracted from raw video frames by a pretrained convolutional neural network to train a recurrent neural network classifier. We evaluate the classifier on two benchmark rodent datasets and one octopus dataset. We show that it achieves high accuracy, requires little training data, and surpasses both human agreement and most comparable existing methods. We also create a confidence score for classifier output, and show that our method provides an accurate estimate of classifier performance and reduces the time required by human annotators to review and correct automatically-produced annotations. We release our system and accompanying annotation interface as an open-source MATLAB toolbox.

show abstract

Section: Resultsmentioning

confidence: 96%

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

Harris

Finn

Kieseler

et al. 2023

Sci Rep

View full text Add to dashboard Cite

show abstract

“…This indicates that the multi-head attention mechanism is useful for recognizing the action, and the proposed MAT-EffNet is a competitive network for action recognition. [12] 63.3 -I3D-RGB [12] 72.1 90.3% ARTNet [52] 70.7 89.3% MoViNet-A5 [66] 71.7 -VidTr-L [67] 70.2 89% MAT-EffNet 72.6 90.8% LRCN [38] RGB + optical flow 82.9 -C3D [26] RGB only + 3D CNNs 85.2 -IDTs [32] RGB only + 3D CNNs 85.9 57.2% Two-stream [1] RGB + optical flow 88.0 59.4% FSTCN [39] RGB + optical flow 88.1 59.1% P3D-199 [65] RGB + 3D CNNs 89.2 62.9% TDD [34] RGB + optical flow 90.3 63.2% STS-network [17] RGB + optical flow + others 90.1 62.4% R-M3D [11] RGB only + 3D CNNs 93.2 65.4% STDAN + RGB difference [58] RGB + optical flow + others 91.0 60.4% TSN Corrnet [55] RGB + optical flow 94.4 70.6% MSM-ResNets [56] RGB + optical flow + others 93.5 66.7% R-STAN-50 [68] RGB + optical flow 91.…”

Section: Exploration Of Mat-effnet On the Kinetics-400 Datasetmentioning

confidence: 99%

Multi-head attention-based two-stream EfficientNet for action recognition

Zhou

Ma²,

Ji³

et al. 2022

Multimedia Systems

View full text Add to dashboard Cite

Recent years have witnessed the popularity of using two-stream convolutional neural networks for action recognition. However, existing two-stream convolutional neural network-based action recognition approaches are incapable of distinguishing some roughly similar actions in videos such as sneezing and yawning. To solve this problem, we propose a Multi-head Attention-based Two-stream EfficientNet (MAT-EffNet) for action recognition, which can take advantage of the efficient feature extraction of EfficientNet. The proposed network consists of two streams (i.e., a spatial stream and a temporal stream), which first extract the spatial and temporal features from consecutive frames by using EfficientNet. Then, a multi-head attention mechanism is utilized on the two streams to capture the key action information from the extracted features. The final prediction is obtained via a late average fusion, which averages the softmax score of spatial and temporal streams. The proposed MAT-EffNet can focus on the key action information at different frames and compute the attention multiple times, in parallel, to distinguish similar actions. We test the proposed network on the UCF101, HMDB51 and Kinetics-400 datasets. Experimental results show that the MAT-EffNet outperforms other state-of-the-art approaches for action recognition.

show abstract

“…Y. Li et al [43] proposed a novel spatio-temporal based deep residual neural network via categorized attentions (STDRN-HA) for human video event detection and classification. In [44], Q. Meng et al proposed the Support Vector Machine SVM classification based novel features extraction method for event classification and detection, however, single SVM is the reason for the system's low accuracy. In [45], S. Sun et al developed a guided optical flow feature extraction approach via Convolutional Neural Network (CNN) for human event detection.…”

Section: Conflicts Of Interestmentioning

confidence: 99%

Human Posture Estimation and Sustainable Events Classification via Pseudo-2D Stick Model and K-ary Tree Hashing

2020

View full text Add to dashboard Cite

This paper suggests that human pose estimation (HPE) and sustainable event classification (SEC) require an advanced human skeleton and context-aware features extraction approach along with machine learning classification methods to recognize daily events precisely. Over the last few decades, researchers have found new mechanisms to make HPE and SEC applicable in daily human life-log events such as sports, surveillance systems, human monitoring systems, and in the education sector. In this research article, we propose a novel HPE and SEC system for which we designed a pseudo-2D stick model. To extract full-body human silhouette features, we proposed various features such as energy, sine, distinct body parts movements, and a 3D Cartesian view of smoothing gradients features. Features extracted to represent human key posture points include rich 2D appearance, angular point, and multi-point autocorrelation. After the extraction of key points, we applied a hierarchical classification and optimization model via ray optimization and a K-ary tree hashing algorithm over a UCF50 dataset, an hmdb51 dataset, and an Olympic sports dataset. Human body key points detection accuracy for the UCF50 dataset was 80.9%, for the hmdb51 dataset it was 82.1%, and for the Olympic sports dataset it was 81.7%. Event classification for the UCF50 dataset was 90.48%, for the hmdb51 dataset it was 89.21%, and for the Olympic sports dataset it was 90.83%. These results indicate better performance for our approach compared to other state-of-the-art methods.

show abstract

Action Recognition Using Form and Motion Modalities

Cited by 14 publications

References 44 publications

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

Multi-head attention-based two-stream EfficientNet for action recognition

Human Posture Estimation and Sustainable Events Classification via Pseudo-2D Stick Model and K-ary Tree Hashing

Contact Info

Product

Resources

About