Predicting skeleton trajectories using a Skeleton-Transformer for video anomaly detection

Pang, Wenfeng; He, Qianhua; Li, Yanxiong

doi:10.1007/s00530-022-00915-9

Cited by 19 publications

(10 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From Table 1, it can be observed that the performance of our STEGT‐AE model on the ShanghaiTech and HR‐ST datasets surpasses several methods based on pixel‐level appearance features [6, 7, 10, 11, 20, 22, 39], optical flow features [17, 40], and their combination [9, 19]. Furthermore, as shown in Table 1, our model outperforms existing methods that utilise pose features [2, 13, 16, 24, 25, 41–43]. Specifically, compared to the ST‐GCN‐based prediction network Normal graph [24], our model achieves a 5.43% and 4.29% improvement in frame‐level AUC on the ShanghaiTech and HR‐ST datasets, respectively.…”

Section: Methodsmentioning

confidence: 80%

“…Specifically, compared to the ST‐GCN‐based prediction network Normal graph [24], our model achieves a 5.43% and 4.29% improvement in frame‐level AUC on the ShanghaiTech and HR‐ST datasets, respectively. Compared to the Skeleton‐transformer method based on the Transformer model [43], the AUC value of our model is improved by 3.14% on the HR‐ST dataset. Compared to STGCAE‐LSTM [25], which also follows a single‐encoder dual‐decoder architecture, the frame‐level AUC values of our model are improved by 3.93% and 3.59% on the ShanghaiTech and HR‐ST datasets, respectively.…”

Section: Methodsmentioning

confidence: 97%

See 1 more Smart Citation

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

Zhu,

Wei,

2023

IET Computer Vision

View full text Add to dashboard Cite

Due to the robustness of skeleton data to human scale, illumination changes, dynamic camera views, and complex backgrounds, great progress has been made in skeleton‐based video anomaly detection in recent years. The spatio‐temporal graph convolutional network has been proven to be effective in modelling the spatio‐temporal dependencies of non‐Euclidean data such as human skeleton graphs, and the autoencoder based on this basic unit is widely used to model sequence features. However, due to the limitations of the convolution kernel, the model cannot capture the correlation between non‐adjacent joints, and it is difficult to deal with long‐term sequences, resulting in an insufficient understanding of behaviour. To address this issue, this paper applies the Transformer to the human skeleton and proposes the Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder (STEGT‐AE) to improve the capability of modelling. In addition, the multi‐memory model with skip connections is employed to provide different levels of coding features, thereby enhancing the ability of the model to distinguish similar heterogeneous behaviours. Furthermore, the STEGT‐AE has a single encoder‐double decoder architecture, which can improve the detection performance by the combining reconstruction and prediction error. The experimental results show that performances of STEGT‐AE is significantly better than other advanced algorithms on four baseline datasets.

show abstract

Section: Methodsmentioning

confidence: 80%

Section: Methodsmentioning

confidence: 97%

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

Zhu,

Wei,

2023

IET Computer Vision

View full text Add to dashboard Cite

show abstract

“…As can be seen in Table I and II, the existing skeletal video anomaly detection methods and available datasets focus towards detecting irregular body postures [16], and anomalous human actions [30] in mostly outdoor settings, and not in proper healthcare settings, such as personal homes and longterm care homes. This a gap towards real world deployment, as there is a need to extend the scope of detecting anomalous behaviours using skeletons to in-home and care home settings, where privacy is a very important concern.…”

Section: Discussionmentioning

confidence: 99%

“…Fan et al [29] proposed a GRU feedforward network that was trained to predict the next skeleton using past skeleton sequences and a loss function that incorporated the range and speed of the predicted skeletons. Pang et al [30] proposed a skeleton transformer to predict future pose components in video frames and considered error between predicted pose components and corresponding expected values as anomaly score. They applied a multi-head self-attention module to capture long-range dependencies between arbitrary pairwise pose components and the temporal convolutional layer to concentrate on local temporal information.…”

Section: B Prediction Approachesmentioning

confidence: 99%

Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

Mishra¹,

Mihailidis²,

Khan³

2023

Preprint

View full text Add to dashboard Cite

The existing methods for video anomaly detection mostly utilize videos containing identifiable facial and appearance-based features. The use of videos with identifiable faces raises privacy concerns, especially when used in a hospital or community-based setting. Appearance-based features can also be sensitive to pixel-based noise, straining the anomaly detection methods to model the changes in the background and making it difficult to focus on the actions of humans in the foreground. Structural information in the form of skeletons describing the human motion in the videos is privacy-protecting and can overcome some of the problems posed by appearance-based features. In this paper, we present a survey of privacy-protecting deep learning anomaly detection methods using skeletons extracted from videos. We present a novel taxonomy of algorithms based on the various learning approaches. We conclude that skeleton-based approaches for anomaly detection can be a plausible privacyprotecting alternative for video anomaly detection. Lastly, we identify major open research questions and provide guidelines to address them.

show abstract

“…In this section, the experimental results of the model in this paper are compared with the existing abnormal behavior detection models with the same indicator on three datasets. In the comparison models, unmasking [28] was the model to learn the efective classifer through sliding windows, the level set method [29] was the model that used horizontal set detection to extract image descriptors, SRNN [32] and sRNN-AE [35] used RNN as the basic model, GAN_pred [33] used GAN combined with U-Net, PST [36] used pose components for abnormal behavior detection, and others were variant models based on autoencoder and U-Net. Table 3 shows the results of the proposed model and the comparison models on three datasets.…”

Section: Quantitative Experimentsmentioning

confidence: 99%

A New Method of Pedestrian Abnormal Behavior Detection Based on Attention Guidance

Huang

2022

Advances in Multimedia

View full text Add to dashboard Cite

In public places, some behavior that violates public order and endangers public safety is defined as abnormal behavior. Moreover, it is a necessary auxiliary means to maintain public order and safety by detecting abnormal behavior in a large number of surveillance videos. However, due to the small proportion of abnormal behavior in video data, the extreme imbalance of data seriously restricts the effectiveness of detection. So, weakly supervised learning has become the most suitable and effective detection method. However, existing weakly supervised methods rarely take the locality and slightness of abnormal behavior into account and ignore the details of extracted features. Based on this, an attention-directed abnormal behavior detection model is proposed. In the two common prediction and reconstruction abnormal behavior detection methods based on weak supervision, suitable attention mechanisms are introduced, respectively, and two corresponding attention-directed networks are proposed. In addition, aiming at the problem of inaccurate thresholds for abnormal behavior division, the loss function of the model is improved and a new abnormal behavior evaluation method is proposed. Experiments were carried out on three classical datasets (the USCD Ped1, USCD Ped2, and CUHK Avenue dataset) for abnormal behavior detection. The best results for the area under the curve (AUC) indicator reached 82.7%, 94.5%, and 87.3%, respectively, which are better than many existing literature results.

show abstract

Predicting skeleton trajectories using a Skeleton-Transformer for video anomaly detection

Cited by 19 publications

References 28 publications

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

A Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder embedded pose for anomaly detection

Skeletal Video Anomaly Detection using Deep Learning: Survey, Challenges and Future Directions

A New Method of Pedestrian Abnormal Behavior Detection Based on Attention Guidance

Contact Info

Product

Resources

About