Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition

Qin, Zhenyue; Liu, Yang; Pan, Ji Sheng; Kim, Dongwoo; Wang, Lei; McKay, Bob; Anwar, Saeed; Gedeon, Tom

doi:10.48550/arxiv.2105.01563

Cited by 9 publications

(12 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Thirty-Sixth AAAI Conference on Artificial Intelligence property / method (group) Fan et al 2020;Yang et al 2020b;Li et al 2019b;Qin et al 2021;Chen et al 2021a;Plizzari, Cannici, and Matteucci 2020;Shi et al 2020a,b;Lei et al 2019), G 3 : (Li, Zhang, and Li 2020;Pan, Chen, and Ortega 2021;Cho et al 2020;Peng et al 2020;Si et al 2018;Song et al 2017a;Xie et al 2018;Huang et al 2020;Si et al 2019).…”

Section: (C) Top Row) the Static Topological Connectionsmentioning

confidence: 99%

“…2s-GCN (Lei et al 2019) further adapts the adjacency matrix to model the learnable dynamic intensity of the joints connection using an embedding function. 2s-GCN also popularizes the use of multi-stream inputs, such as joint, bone, joint motion, bone motion, angular, etc., for skeleton-based action recognition (Lei et al 2019;Shi et al 2020b;Qin et al 2021). A common drawback of these works is that they use the same adjacency matrices for different inputs, and the adjacency matrices they use only model the joints' physical topology connection, which limits their adaptivity to largely different and dynamic actions.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Ke¹,

Peng²,

Lyu³

2022

AAAI

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have been widely used to model the high-order dynamic dependencies for skeleton-based action recognition. Most existing approaches do not explicitly embed the high-order spatio-temporal importance to joints’ spatial connection topology and intensity, and they do not have direct objectives on their attention module to jointly learn when and where to focus on in the action sequence. To address these problems, we propose the To-a-T Spatio-Temporal Focus (STF), a skeleton-based action recognition framework that utilizes the spatio-temporal gradient to focus on relevant spatio-temporal features. We first propose the STF modules with learnable gradient-enforced and instance-dependent adjacency matrices to model the high-order spatio-temporal dynamics. Second, we propose three loss terms defined on the gradient-based spatio-temporal focus to explicitly guide the classifier when and where to look at, distinguish confusing classes, and optimize the stacked STF modules. STF outperforms the state-of-the-art methods on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets in all 15 settings over different views, subjects, setups, and input modalities, and STF also shows better accuracy on scarce data and dataset shifting settings.

show abstract

Section: (C) Top Row) the Static Topological Connectionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Ke¹,

Peng²,

Lyu³

2022

AAAI

View full text Add to dashboard Cite

show abstract

“…-Encoding semantic information: [53] -Transforming geometric coordinates: [54], [49], [14], [55] -Mitigating overfitting problems: [31] -Designing new features: [56], [57] In the rest of this section, we dive into details of a proportion of existing approaches listed above.…”

Section: Extracting Spatial Featuresmentioning

confidence: 99%

ANUBIS: Skeleton Action Recognition Dataset, Review, and Benchmark

Qin¹,

Liu²,

Perera³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Skeleton-based action recognition, as a subarea of action recognition, is swiftly accumulating attention and popularity. The task is to recognize actions performed by human articulation points. Compared with other data modalities, 3D human skeleton representations have extensive unique desirable characteristics, including succinctness, robustness, racial-impartiality, and many more. We aim to provide a roadmap for new and existing researchers a on the landscapes of skeleton-based action recognition for new and existing researchers. To this end, we present a review in the form of a taxonomy on existing works of skeleton-based action recognition. We partition them into four major categories: (1) datasets; (2) extracting spatial features; (3) capturing temporal patterns; (4) improving signal quality. For each method, we provide concise yet informatively-sufficient descriptions. To promote more fair and comprehensive evaluation on existing approaches of skeleton-based action recognition, we collect ANUBIS, a large-scale human skeleton dataset. Compared with previously collected dataset, ANUBIS are advantageous in the following four aspects: (1) employing more recently released sensors; (2) containing novel back view; (3) encouraging high enthusiasm of subjects; (4) including actions of the COVID pandemic era. Using ANUBIS, we comparably benchmark performance of current skeleton-based action recognizers. At the end of this paper, we outlook future development of skeleton-based action recognition by listing several new technical problems. We believe they are valuable to solve in order to commercialize skeleton-based action recognition in the near future.

show abstract

“…2s-GCN (Shi et al 2019) further adapts the adjacency matrix to model the learnable dynamic intensity of the joints connection using an embedding function. 2s-GCN also popularizes the use of multi-stream inputs, such as joint, bone, joint motion, bone motion, angular, etc., for skeletonbased action recognition (Shi et al 2019(Shi et al , 2020bQin et al 2021). A common drawback of these works is that they use the same adjacency matrices for different inputs, and the adjacency matrices they use only model the joints' physical topology connection, which limits their adaptivity to largely different and dynamic actions.…”

Section: Related Workmentioning

confidence: 99%

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Ke¹,

Peng²,

Lyu³

2022

Preprint

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have been widely used to model the high-order dynamic dependencies for skeleton-based action recognition. Most existing approaches do not explicitly embed the high-order spatio-temporal importance to joints' spatial connection topology and intensity, and they do not have direct objectives on their attention module to jointly learn when and where to focus on in the action sequence. To address these problems, we propose the To-a-T Spatio-Temporal Focus (STF), a skeleton-based action recognition framework that utilizes the spatio-temporal gradient to focus on relevant spatio-temporal features. We first propose the STF modules with learnable gradient-enforced and instance-dependent adjacency matrices to model the high-order spatio-temporal dynamics. Second, we propose three loss terms defined on the gradientbased spatio-temporal focus to explicitly guide the classifier when and where to look at, distinguish confusing classes, and optimize the stacked STF modules. STF outperforms the state-of-the-art methods on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets in all 15 settings over different views, subjects, setups, and input modalities, and STF also shows better accuracy on scarce data and dataset shifting settings.

show abstract

Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition

Cited by 9 publications

References 31 publications

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

ANUBIS: Skeleton Action Recognition Dataset, Review, and Benchmark

Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition

Contact Info

Product

Resources

About