Revisiting Skeleton-based Action Recognition

Autism Spectrum Disoder (ASD) is a neurodevelopmental disorder characterized by (a) persistent deficits in social communication and interaction, and (b) presence of restrictive, repetitive patterns of behaviours, interests or activities. The stereotyped repetitive behaviours are also referred to as stimming behaviours. We propose a deep learning based approach to automatically predict a child's stimming behaviours from videos recorded in unconstrained conditions. The child's region in the video is tracked and its skeletal joints are derived using the pose estimator. The heatmap representation of skeletal joints and the raw video signals are used as inputs to the two pathways of the RGBPose-SlowFast deep network to model stimming behaviours. The proposed model is evaluated using the publicly available Self-Stimulatory Behaviour Dataset (SSBD) of stimming behaviours. The generalization ability of the model is validated using the Autism dataset containing child's motor actions. Our experiments demonstrate state-of-the-art results on both datasets.

show abstract

“…The primary goal of the proposed work is to detect a child's stimming behaviour using RGBPose-SlowFast Network [17].…”

Section: Proposed Methods For Detecting a Child's Stimming Behaviourmentioning

confidence: 99%

Detecting A Child’s Stimming Behaviours for Autism Spectrum Disorder Diagnosis using Rgbpose-Slowfast Network

Pandian

Rajagopalan

et al. 2022

2022 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

show abstract

“…Although GCN models have obtained state‐of‐the‐art results on different datasets, GCN models are computationally complex and subject to robustness limitations when working with small datasets [6, 17]. Therefore, in Ref.…”

Section: Related Workmentioning

confidence: 99%

A robust and efficient method for skeleton‐based human action recognition and its application for cross‐dataset evaluation

Nguyen

Pham

et al. 2022

IET Computer Vision

View full text Add to dashboard Cite

Skeleton-based human action recognition has emerged recently thanks to its compactness and robustness to appearance variations. Although impressive results have been obtained in recent years, the performance of skeleton-based action recognition methods has to be improved to be deployed in real-time applications. Recently, a lightweight network structure named Double-feature Double-motion Network (DD-Net) has been proposed for the skeleton-based human action recognition. With high speed, the DD-Net achieves state-of-the-art performance on hand and body actions. The DD-Net could not distinguish actions if they have a weak connection with the global trajectories. However, the DD-Net is suitable for human action recognition where actions strongly correlate to the global trajectories. In this paper, the authors propose TD-Net, an improved version of the DD-Net in which a new branch is added. The new branch takes the normalised coordinates of joints (NCJ) to enrich the spatial information. On five datasets for skeletonbased human activity recognition that are MSR-Action3D, CMDFall, JHMDB, FPHAB, and NTU RGB + D, the TD-Net consistently obtains superior performance compared with the baseline model DD-Net. The proposed method outperforms different state-ofthe-art methods, including both hand-designed and deep learning-based methods on four datasets (MSR-Action3D, CMDFall, JHMDB, and FPHAB). Furthermore, the generalisation of the proposed method is confirmed through cross-dataset evaluation. To illustrate the potential use of the model for real-time human action recognition, the authors have deployed an application on an edge device. The experimental result shows that the application can process up to 40 fps for pose estimation using MediaPipe. It takes only 0.04 ms to recognise an action from skeleton sequences.

show abstract

“…A heatmap based 3D CNN action recognition model, PoseC3D [14] was also introduced. PoseC3D is a variant of 3D CNN [15]- [17] model that uses a 3D or (2+1)D convolutional layer to extract spatiotemporal features.…”

Section: Related Work a Cnn-based Skeleton Action Recognitionmentioning

confidence: 99%

Rank-GCN for Robust Action Recognition

et al. 2022

View full text Add to dashboard Cite

We present a robust skeleton-based action recognition method with graph convolutional network (GCN) that uses the new adjacency matrix, called Rank-GCN. In Rank-GCN, the biggest change from previous approaches is how the adjacency matrix is generated to accumulate features from neighboring nodes by re-defining "adjacency." The new adjacency matrix, which we call the rank adjacency matrix, is generated by ranking all the nodes according to metrics including the Euclidean distance from the nodes of interest, whereas the previous GCNs methods used only 1-hop neighboring nodes to construct adjacency. By adopting the rank adjacency matrix, we find not only performance improvements but also robustness against swapping, location shifting and dropping of certain nodes. The fact that the human-made rank adjacency matrix wins against the deep-learning-based matrix, implies that there are still some parts that need touch of humans. We expect our Rank-GCN can make performance improvements especially when the predicted human joints are less accurate and unstable.

show abstract

Revisiting Skeleton-based Action Recognition

Cited by 407 publications

References 38 publications

Detecting A Child’s Stimming Behaviours for Autism Spectrum Disorder Diagnosis using Rgbpose-Slowfast Network

Detecting A Child’s Stimming Behaviours for Autism Spectrum Disorder Diagnosis using Rgbpose-Slowfast Network

A robust and efficient method for skeleton‐based human action recognition and its application for cross‐dataset evaluation

Rank-GCN for Robust Action Recognition

Contact Info

Product

Resources

About