Multi-head attention-based two-stream EfficientNet for action recognition

Zhou, Aoying; Ma, Yujun; Ji, Wanting; Zong, Ming; Yang, Ping; Wu, Min; Liu, Mingzhe

doi:10.1007/s00530-022-00961-3

Cited by 23 publications

(7 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Semi-supervised temporal gradient learning [57] 2022 75.9 BS-2SCN [41] 2022 71.3 ViT + Multi Layer LSTM [45] 2022 73.7 MAT-EffNet [58] 2023 70.9 DA-R3DCNN (Proposed) 2023 82.5…”

Section: Methods Year Accuracy (%)mentioning

confidence: 99%

“…Among the comparative methods, Multi-task hierarchical clustering [33] achieved the lowest accuracy of 51.4% on the HMDB51 dataset. Other comparative methods included STPP + LSTM [46], TSN [48], Deep autoencoder [35], TS-LSTM + temporal-inception [50], HATNet [51], Correlational CNN + LSTM [52], STDAN [53], DB-LSTM + SSPF [54], DS-GRU [39], TCLC [55], Semi-supervised temporal gradient learning [57], BS-2SCN [41], ViT + Multi Layer LSTM [45], and MAT-EffNet [58]. These methods achieved accuracies of 70.5%, 72.2%, 70.7%, 58.6%, 70.3%, 69.0%, 74.8%, 66.2%, 56.5%, 75.1%, 72.3%, 71.5%, 75.9%, 71.3%, 73.7%, and 70.9%, respectively.…”

Section: Methods Year Accuracy (%)mentioning

confidence: 99%

See 1 more Smart Citation

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

Ullah,

Munir

2023

Algorithms

View full text Add to dashboard Cite

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

show abstract

“…Semi-supervised temporal gradient learning [57] 2022 75.9 BS-2SCN [41] 2022 71.3 ViT + Multi Layer LSTM [45] 2022 73.7 MAT-EffNet [58] 2023 70.9 DA-R3DCNN (Proposed) 2023 82.5…”

Section: Methods Year Accuracy (%)mentioning

confidence: 99%

Section: Methods Year Accuracy (%)mentioning

confidence: 99%

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

Ullah,

Munir

2023

Algorithms

View full text Add to dashboard Cite

show abstract

“…It extracts static features from partial facial images and dynamic features from partial facial optical flow, feeding them into a dual-stream neural network for feature fusion and classification, showcasing excellent performance. Zhou et al [46] harnessed EfficientNet's efficient feature extraction abilities to separately extract spatial and temporal features of consecutive video frames from spatial and temporal flows. They then employed a multi-head attention mechanism to capture pivotal action details, facilitating action recognition using the amalgamated features.…”

Section: Deep Multi-path Networkmentioning

confidence: 99%

Chrysanthemum Classification Method Integrating Deep Visual Features from both the Front and Back Sides

Chen,

Yang,

Yan

et al. 2024

Preprint

View full text Add to dashboard Cite

Chrysanthemum morifolium Ramat(hereinafter referred to as Chrysanthemum) is one of the most beloved and economically valuable Chinese herbal crops, which contains abundant medicinal ingredients and wide application prospects. Therefore, identifying the classification and origin of Chrysanthemum is important for producers, consumers, and market regulators. The existing Chrysanthemum classification methods mostly rely on visual subjective identification, are time-consuming, and always need high equipment costs. A novel method is proposed to accurately identify the Chrysanthemum classification in a swift, non-invasive, and non-contact way. The proposed method is based on the fusion of deep visual features of both the front and back sides. Firstly, the different Chrysanthemums images are collected and labeled with origins and classifications. Secondly, the background area with less available information is removed by image preprocessing. Thirdly, a two-stream feature extraction network is designed with two inputs which are the preprocessed front and back Chrysanthemum images. Meanwhile, the incorporation of single-path residual connections and cross-path residual connections is employed to extend the receptive field of the network and fully fusion the features from both the front and back sides. Experimental results demonstrate the proposed method outperforms existing methods, which gain higher accuracy and stability. The proposed method presents an effective and dependable solution for identifying the classification and origin of Chrysanthemum. Code and data are available at https://github.com/dart-into/CCMIFB.

show abstract

“…The two-stream convolutional network [16] was proposed, which had two feature extraction networks, one using optical flow and the other using RGB. Along with CNN, a number of much stronger networks appeared, such as TSN [17], TSM [18] and R(2+1)D [19] etc [20,21]. Although 2D-CNN has smaller model parameters and higher accuracy for image data, they cannot obtain more effective information of video data.…”

Section: Related Workmentioning

confidence: 99%

You watch once more: a more effective CNN architecture for video spatio-temporal action localization

Qin,

Chen,

Ben

et al. 2024

Multimedia Systems

View full text Add to dashboard Cite

The task of spatio-temporal action localization (STAL) needs to detect the action and position of individuals in the scene. Many works focus on how to improve the accuracy, but they usually ignore inference speed and practical applications. To address the above problems, we propose a new end-to-end spatio-temporal action localization network called You Watch Once More (YWOM). In this work, there are three measures proposed to improve the accuracy of positioning and recognition while guaranteeing the inference speed. First, a new feature fusion mechanism based on frequency channel attention (FCA) is proposed, which can effectively fuse the features extracted by different backbones. In addition, a new loss function is proposed to speed up the regression and convergence of the bounding box. Specifically, the SIOU regression loss function instead of the smooth L1 loss function is applied to help the model converge stably. Moreover, a lateral connection mechanism is designed to apply more backbones to our network structure. The experimental results demonstrate that YWOM can achieve online inference speed and has good performance of spatio-temporal action localization tasks. YWOM has superiority over other related works including YOWO on the UCF101-24 dataset. The frame-mAP and the video-mAP (0.2) are improved by 4.23% and 1.63% respectively.

show abstract

Multi-head attention-based two-stream EfficientNet for action recognition

Cited by 23 publications

References 72 publications

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

Chrysanthemum Classification Method Integrating Deep Visual Features from both the Front and Back Sides

You watch once more: a more effective CNN architecture for video spatio-temporal action localization

Contact Info

Product

Resources

About