Skeleton-based Action Recognition via Temporal-Channel Aggregation

Wang, Shengqin; Zhang, Yongji; Wei, Fenglin; Wang, Kai; Zhao, Minghao; Yu, Junsheng

doi:10.48550/arxiv.2205.15936

Cited by 6 publications

(10 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed module is plug-and-play and compatible with most GCN-based backbones. To examine its universality, we apply it to 5 widely used GCN-based backbones [5,17,26,35,38] and evaluate them on the X-Sub and X-Set of NTU RGB+D 120 dataset. For fair comparisons, we reimplement them and use the same data preprocessing without the multi-stream fusion.…”

Section: Combined With Other Backbonesmentioning

confidence: 99%

See 1 more Smart Citation

Learning Discriminative Representations for Skeleton Based Action Recognition

Zhou¹,

Liu²,

Wang³

2023

Preprint

View full text Add to dashboard Cite

Human action recognition aims at classifying the category of human action from a segment of a video. Recently, people have dived into designing GCN-based models to extract features from skeletons for performing this task, because skeleton representations are much more efficient and robust than other modalities such as RGB frames. However, when employing the skeleton data, some important clues like related items are also discarded. It results in some ambiguous actions that are hard to be distinguished and tend to be misclassified. To alleviate this problem, we propose an auxiliary feature refinement head (FR Head), which consists of spatial-temporal decoupling and contrastive feature refinement, to obtain discriminative representations of skeletons. Ambiguous samples are dynamically discovered and calibrated in the feature space. Furthermore, FR Head could be imposed on different stages of GCNs to build a multi-level refinement for stronger supervision. Extensive experiments are conducted on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets. Our proposed models obtain competitive results from state-of-the-art methods and can help to discriminate those ambiguous samples. Codes are available at https://github.com/zhysora/FR-Head.

show abstract

Section: Combined With Other Backbonesmentioning

confidence: 99%

“…It is noted that most of the state-of-theart methods employ a multi-stream fusion framework. For a fair comparison, we follow the same framework as [5,35]. We make a fusion with the results from four modalities including joint, bone, joint motion, and bone motion as the final report result.…”

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

Learning Discriminative Representations for Skeleton Based Action Recognition

Zhou¹,

Liu²,

Wang³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Literature [7] and literature [10] have computational costs similar to that of this paper, but the accuracy is 5.3% and 0.6% lower than that of this paper, respectively. Literature [11][12][13][14][15][16] has a higher performance in recognition accuracy, but also has a larger amount of computation and parameters. Therefore, the method proposed in this paper balances the three performance indexes of calculation cost, parameter number and recognition accuracy well, and ensures a high recognition accuracy while reducing parameter number and calculation cost.…”

Section: Comparison With Other Action Recognition Algorithmsmentioning

confidence: 99%

Lightweight skeleton-based action recognition research

Wang

2023

3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023)

View full text Add to dashboard Cite

Action recognition is a cutting-edge research direction in the field of computer vision, widely used in video surveillance, video retrieval, human-computer interaction, and other fields. However, existing action recognition algorithms improve prediction accuracy at the cost of having too many model parameters and high computational complexity, which limits their practical application. In order to address this issue and improve recognition accuracy with higher computational efficiency, we propose a dense residual 3D depth-separable convolutional network model based on channel attention mechanism. By extending the depth separable convolution to 3D, the model can maintain high accuracy while greatly reducing the number of model parameters and operational complexity. The dense residual structure is introduced to enhance the training ability of the network, improve feature extraction, and further improve network performance by learning the correlation between channels through the channel attention mechanism. Compared with other action recognition algorithms on two datasets, NTU RGB-D and Fall Dataset Lei2, the proposed action recognition algorithm has better robustness under lightweight condition, with 415.9 MFLOPs of computation, 205.4k of parameters, and 90.5% of CS, 95.5% of CV and 95.97% of accuracy, respectively. The proposed algorithm has a high recognition accuracy and greatly reduces the calculation cost, which can provide better performance for practical applications.

show abstract

“…Zhou et al [ 26 ] introduced a graph attention block based on Convolutional Block Attention Module (CBAM), which is used to calculate the semantic correlation between any two joints. TCA-GCN [ 27 ] used an MS-CAM attention fusion mechanism to solve the problem of the contextual aggregation of skeletal features. However, at present, the local and global context aggregation of skeletal features and the integration of initial features are still a major difficulty, and how to effectively integrate initial features into high-dimensional features needs further research.…”

Section: Related Workmentioning

confidence: 99%

Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Zhang

et al. 2023

Sensors

View full text Add to dashboard Cite

Graph convolutional networks are widely used in skeleton-based action recognition because of their good fitting ability to non-Euclidean data. While conventional multi-scale temporal convolution uses several fixed-size convolution kernels or dilation rates at each layer of the network, we argue that different layers and datasets require different receptive fields. We use multi-scale adaptive convolution kernels and dilation rates to optimize traditional multi-scale temporal convolution with a simple and effective self attention mechanism, allowing different network layers to adaptively select convolution kernels of different sizes and dilation rates instead of being fixed and unchanged. Besides, the effective receptive field of the simple residual connection is not large, and there is a great deal of redundancy in the deep residual network, which will lead to the loss of context when aggregating spatio-temporal information. This article introduces a feature fusion mechanism that replaces the residual connection between initial features and temporal module outputs, effectively solving the problems of context aggregation and initial feature fusion. We propose a multi-modality adaptive feature fusion framework (MMAFF) to simultaneously increase the receptive field in both spatial and temporal dimensions. Concretely, we input the features extracted by the spatial module into the adaptive temporal fusion module to simultaneously extract multi-scale skeleton features in both spatial and temporal parts. In addition, based on the current multi-stream approach, we use the limb stream to uniformly process correlated data from multiple modalities. Extensive experiments show that our model obtains competitive results with state-of-the-art methods on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.

show abstract

Skeleton-based Action Recognition via Temporal-Channel Aggregation

Cited by 6 publications

References 38 publications

Learning Discriminative Representations for Skeleton Based Action Recognition

Learning Discriminative Representations for Skeleton Based Action Recognition

Lightweight skeleton-based action recognition research

Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition

Contact Info

Product

Resources

About