Learning Dynamic Spatio-Temporal Relations for Human Activity Recognition

Liu, Zhenyu; Yao, Yaqiang; Liu, Yan; Zhu, Yuening; Tao, Zhenchao; Wang, Lei; Yang, Feng

doi:10.1109/access.2020.3009136

Cited by 8 publications

(4 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The two methods [14] of human activity recognition as shown in Fig. 2 have drawbacks of their own like the vision-based human activity recognition is easily impacted by external factors including lighting condition, clothing color, image background, and so on.…”

Section: Introductionmentioning

confidence: 99%

Human Stress Recognition by Correlating Vision and EEG Data

Praveenkumar¹,

Karthick²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

Because stress has such a powerful impact on human health, we must be able to identify it automatically in our everyday lives. The human activity recognition (HAR) system use data from several kinds of sensors to try to recognize and evaluate human actions automatically recognize and evaluate human actions. Using the multimodal dataset DEAP (Database for Emotion Analysis using Physiological Signals), this paper presents deep learning (DL) technique for effectively detecting human stress. The combination of vision-based and sensor-based approaches for recognizing human stress will help us achieve the increased efficiency of current stress recognition systems and predict probable actions in advance of when fatal. Based on visual and EEG (Electroencephalogram) data, this research aims to enhance the performance and extract the dominating characteristics of stress detection. For the stress identification test, we utilized the DEAP dataset, which included video and EEG data. We also demonstrate that combining video and EEG characteristics may increase overall performance, with the suggested stochastic features providing the most accurate results. In the first step, CNN (Convolutional Neural Network) extracts feature vectors from video frames and EEG data. Feature Level (FL) fusion that combines the features extracted from video and EEG data. We use XGBoost as our classifier model to predict stress, and we put it into action. The stress recognition accuracy of the proposed method is compared to existing methods of Decision Tree (DT), Random Forest (RF), AdaBoost, Linear Discriminant Analysis (LDA), and K-Nearest Neighborhood (KNN). When we compared our technique to existing state-of-the-art approaches, we found that the suggested DL methodology combining multimodal and heterogeneous inputs may improve stress identification.

show abstract

Section: Introductionmentioning

confidence: 99%

Human Stress Recognition by Correlating Vision and EEG Data

Praveenkumar¹,

Karthick²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

show abstract

“…As shown in Tables 3, 5 and 6 [27] 93.3 *koppula et al [24] 80.6 *Tayyub et al [45] 95.2 Sanou et al [36] 86.4 GLIDN (ours) 88.54…”

Section: Are Contextual Views Of Humans and Objects Important?mentioning

confidence: 94%

Distillation of Human-Object Interaction Contexts for Action Recognition

Almushyti¹,

Li²

2021

Preprint

View full text Add to dashboard Cite

Modeling spatial-temporal relations is imperative for recognizing human actions, especially when a human is interacting with objects, while multiple objects appear around the human differently over time. Most existing action recognition models focus on learning overall visual cues of a scene but disregard informative fine-grained features, which can be captured by learning human-object relationships and interactions. In this paper, we learn human-object relationships by exploiting the interaction of their local and global contexts. We hence propose the Global-Local Interaction Distillation Network (GLIDN), learning human and object interactions through space and time via knowledge distillation for fine-grained scene understanding. GLIDN encodes humans and objects into graph nodes and learns local and global relations via graph attention network [52]. The local context graphs learn the relation between humans and objects at a frame level by capturing their cooccurrence at a specific time step. The global relation graph is constructed based on the video-level of human and object interactions, identifying their long-term relations throughout a video sequence. More importantly, we investigate how knowledge from these graphs can be distilled to their counterparts for improving human-object interaction (HOI) recognition. We evaluate our model by conducting comprehensive experiments on two datasets including Charades [40] and CAD-120 [24] datasets. We have achieved better results than the baselines and counterpart approaches.

show abstract

“…Also, our STIT [1]. NOTE THAT [52], [53], [1] AND [54] HAVE EMPLOYED ADDITIONAL SKELETON OR DEPTH INFORMATION.…”

Section: B Experiments On the Charades Dataset 1) Implementation Detailsmentioning

confidence: 99%

“…Accuracy% Wang et al [52] 81.2 Liu et al [53] 93.3 koppula et al [1] 80.6 Tayyub et al [54] 95.2 Sanou et al [50] 93.6 STIT (ours) 95.93 model can be incorporated with any backbone model rather than I3D without end-to-end training. As a result, our STIT model with Slowfast 16 x 8 surpasses its baseline.…”

Section: Modelmentioning

confidence: 99%

STIT: Spatio-Temporal Interaction Transformers for Human-Object Interaction Recognition in Videos

Almushyti

2022

2022 26th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Recognizing human-object interactions is challenging due to their spatio-temporal changes. We propose the Spatio-Temporal Interaction Transformer-based (STIT) network to reason such changes. Specifically, spatial transformers learn humans and objects context at specific frame time. Temporal transformer then learns the relations at a higher level between spatial context representations at different time steps, capturing longterm dependencies across frames. We further investigate multiple hierarchy designs in learning human interactions. We achieved superior performance on Charades, Something-Something v1 and CAD-120 datasets, comparing to baseline models without learning human-object relations, or with prior graph-based networks. We also achieved state-of-the-art accuracy of 95.93% on CAD-120 dataset [1] by employing RGB data only.

show abstract

Learning Dynamic Spatio-Temporal Relations for Human Activity Recognition

Cited by 8 publications

References 47 publications

Human Stress Recognition by Correlating Vision and EEG Data

Human Stress Recognition by Correlating Vision and EEG Data

Distillation of Human-Object Interaction Contexts for Action Recognition

STIT: Spatio-Temporal Interaction Transformers for Human-Object Interaction Recognition in Videos

Contact Info

Product

Resources

About