Human Activity Recognition from Wearable Sensor Data Using Self-Attention

Abstract

Human Activity Recognition from body-worn sensor data poses an inherent challenge in capturing spatial and temporal dependencies of time-series signals. In this regard, the existing recurrent or convolutional or their hybrid models for activity recognition struggle to capture spatio-temporal context from the feature space of sensor reading sequence. To address this complex problem, we propose a self-attention based neural network model that foregoes recurrent architectures and utilizes different types of attention mechanisms to generate higher dimensional feature representation used for classification. We performed extensive experiments on four popular publicly available HAR datasets: PAMAP2, Opportunity, Skoda and USC-HAD. Our model achieve significant performance improvement over recent state-of-the-art models in both benchmark test subjects and Leave-one-subject-out evaluation. We also observe that the sensor attention maps produced by our model is able capture the importance of the modality and placement of the sensors in predicting the different activity classes.

TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

“…The proposed TASKED framework was evaluated on the Opportunity, PAMAP2, MHEALTH, and RealDISP datasets. The three evaluation metrics were used to evaluate and compare the proposed method to deep-learningbased state-of-the-art methods including multi-channel timeseries convolutional neural networks (MC-CNN) [10], Deep-ConvLSTM [11], Self-attention activity recognition method [12], METIER model [18], and the previous method "Adversarial CNN" [26]. MC-CNN is a CNN-based model consisting of three convolutional layers, two pooling layers, and two fully connected layers.…”

“…Zeng et al [49] applied the self-attention mechanism on the LSTM networks to highlight the important part of time-series signals. Similarly, Mahmud et al [12] proposed an activity recognition method by employing self-attention, which has reached state-of-the-art results. However, such deep models often require high computation and memory resources.…”

“…Those models have achieved state of the art results in many areas including natural language processing [65], vision [29], and skeleton-based HAR [30]. Other works, such as [12] also use self-attention for wearable sensor-based HAR obtaining state-of-the-art results. Still, their model combines data from the different present sensors in the first layers using 1x1 convolutions, forgoing richer explicit attention across sensors further down the network.…”

“…Recently, deep learning methods have achieved state-of-the-art performance in HAR by using convolutional neural networks (CNN) as in [10] or by combining them with long short-term memory networks (LSTM) as in [11]. Even more recently, self-attention approaches have reached state-of-the-art results [12,13].…”

Contrastive Predictive Coding for Human Activity Recognition

“…Subsequent work on this architecture to improve performance includes temporal attention [55]. More recently, Transformer [75] encoder networks have been utilized for activity recognition [32,50]. They model sequential information via the use of self-attention mechanisms and utilize only dense layers.…”