A Self-Attention Augmented Graph Convolutional Clustering Networks for Skeleton-Based Video Anomaly Behavior Detection

Abstract

In this paper, we propose a new method for detecting abnormal human behavior based on skeleton features using self-attention augment graph convolution. The skeleton data have been proved to be robust to the complex background, illumination changes, and dynamic camera scenes and are naturally constructed as a graph in non-Euclidean space. Particularly, the establishment of spatial temporal graph convolutional networks (ST-GCN) can effectively learn the spatio-temporal relationships of Non-Euclidean Structure Data. However, it only operates on local neighborhood nodes and thereby lacks global information. We propose a novel spatial temporal self-attention augmented graph convolutional networks (SAA-Graph) by combining improved spatial graph convolution operator with a modified transformer self-attention operator to capture both local and global information of the joints. The spatial self-attention augmented module is used to understand the intra-frame relationships between human body parts. As far as we know, we are the first group to utilize self-attention for video anomaly detection tasks by enhancing spatial temporal graph convolution. Moreover, to validate the proposed model, we performed extensive experiments on two large-scale publicly standard datasets (i.e., ShanghaiTech Campus and CUHK Avenue datasets) which reveal the state-of-art performance for our proposed approach when compared to existing skeleton-based methods and graph convolution methods.

Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

“…In the fine-tuning step, the entire network was fine-tuned using a multi-objective loss function, composed of reconstruction loss, prototype generation loss and cluster loss. Later, Liu et al [38] used self-attention augmented graph convolutions for detecting abnormal human behaviours based on skeleton graphs. Skeleton graphs were fed as input to a spatio-temporal selfattention augmented GCAE and latent features were extracted from the encoder part of the trained GCAE.…”

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

“…Markovitz [2] used GCN to learn skeletal joint dependencies for representing behavioural features and performed clustering through soft assignments. Liu [3] extracted local and global features of the skeleton and generated latent vectors for clustering. Luo [24] stacked multilayer ST‐GCN and introduced the Resnet mechanism to detect anomalies by calculating the mean squared error between predicted joints and ground truth.…”

“…These methods are mostly considered as unsupervised learning and can be divided into two types of methods. The first method extracts features from the original video and then uses clustering [1][2][3] or classifiers [4,5] for anomaly detection. The second method reconstructs or predicts the input sequence and calculate the error between the real sequence and the generated sequence.…”

“…Subsequently, many studies have improved ST‐GCN. Similar to skeleton‐based action recognition methods [23], most studies [2–5, 16, 24–26] modelled the spatio‐temporal information of motion targets using the ST‐GCN. Although GCN has strong spatial feature learning capability, it tends to ignore global dependencies, which are crucial for behavioural understanding.…”

A Self-Attention Feature Fusion Model for Rice Pest Detection

“…Figure 1(c The self-attention mechanism [24] is widely used in the field of object detection, it has not been applied to rice pest detection. Liu et al [25] proposed a video detection method for detecting abnormal human behavior. They used a spatial self-attention module, to understand the intra-frame relationship between various parts of the human body, and conducted experiments on large public dataset.…”