Personalized Human Activity Recognition Using Convolutional Neural Networks

Rokni, Seyed Ali; Nourollahi, Marjan; Ghasemzadeh, Hassan

doi:10.1609/aaai.v32i1.12185

Cited by 44 publications

(19 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the most used methods for these types of temporal data is pattern mining to find the frequent action sequences in the data and then determine the activity from the pattern [Liu et al 2016]. Besides, deep neural networks were used for human activity recognition from sensor data [Cheng et al 2018;Rokni, Nourollahi, and Ghasemzadeh 2018]. All of these works used temporal data that are from sensors of an individual user.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Dual Attention-Based Recurrent Neural Networks for Individual and Group Activity Recognition in Games

Ahmad

El-Nasr

Elhamifar

2021

AIIDE

View full text Add to dashboard Cite

We study the problem of simultaneously recognizing complex individual and group activities from spatiotemporal data in games. Recognizing complex player activities is particularly important to understand game dynamics and user behavior having a wide range of applications in game development. To do so, we propose a novel framework by developing a hierarchical dual attention RNN-based method that leverages feature and temporal attention mechanisms in a hierarchical setting for effective discovery of activities using interactions among individuals. We argue that certain activities have dependency on certain features as well as on temporal aspects of the data which can be leveraged by our dual-attention model for recognition. To the best of our knowledge, this work is the first to address activity recognition using spatiotemporal data in games. In addition, we propose using game data as a rich source of obtaining complex group interactions. In this paper, we present two contributions: (1) two annotated game datasets that consist of individual and group activities, (2) our proposed framework improves the state-of-the-art recognition algorithms for spatiotemporal data by experiments on these datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Hierarchical Dual Attention-Based Recurrent Neural Networks for Individual and Group Activity Recognition in Games

Ahmad

El-Nasr

Elhamifar

2021

AIIDE

View full text Add to dashboard Cite

show abstract

“…The authors of [ 16 ] proposed an HAR model with a particular layer with few parameters inserted between every two user-dependent layers of the CNN for personalization. The authors of [ 17 ] proposed to personalize their models with transfer learning. The authors of [ 18 ] proposed a personalized HAR model based on multi-task learning techniques, where each task corresponds to a specific person.…”

Section: Introductionmentioning

confidence: 99%

Federated Meta-Learning with Attention for Diversity-Aware Human Activity Recognition

Shen

Feng

Song

et al. 2023

Sensors

View full text Add to dashboard Cite

The ubiquity of smartphones equipped with multiple sensors has provided the possibility of automatically recognizing of human activity, which can benefit intelligent applications such as smart homes, health monitoring, and aging care. However, there are two major barriers to deploying an activity recognition model in real-world scenarios. Firstly, deep learning models for activity recognition use a large amount of sensor data, which are privacy-sensitive and hence cannot be shared or uploaded to a centralized server. Secondly, divergence in the distribution of sensory data exists among multiple individuals due to their diverse behavioral patterns and lifestyles, which contributes to difficulty in recognizing activity for large-scale users or ’cold-starts’ for new users. To address these problems, we propose DivAR, a diversity-aware activity recognition framework based on a federated Meta-Learning architecture, which can extract general sensory features shared among individuals by a centralized embedding network and individual-specific features by attention module in each decentralized network. Specifically, we first classify individuals into multiple clusters according to their behavioral patterns and social factors. We then apply meta-learning in the architecture of federated learning, where a centralized meta-model learns common feature representations that can be transferred across all clusters of individuals, and multiple decentralized cluster-specific models are utilized to learn cluster-specific features. For each cluster-specific model, a CNN-based attention module learns cluster-specific features from the global model. In this way, by training with sensory data locally, privacy-sensitive information existing in sensory data can be preserved. To evaluate the model, we conduct two data collection experiments by collecting sensor readings from naturally used smartphones annotated with activity information in the real-life environment and constructing two multi-individual heterogeneous datasets. In addition, social characteristics including personality, mental health state, and behavior patterns are surveyed using questionnaires. Finally, extensive empirical results demonstrate that the proposed diversity-aware activity recognition model has a relatively better generalization ability and achieves competitive performance on multi-individual activity recognition tasks.

show abstract

“…Modern supervised learning approaches using convolutional and or recurrent neural networks are increasingly utilized and have demonstrated improvements in classification accuracy over non-neural models [ 1 , 17 , 18 , 19 , 20 , 21 ]. Both non-neural and neural network supervised learning models have been applied to personalized activity recognition [ 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ].…”

Section: Introductionmentioning

confidence: 99%

“…This is generally not feasible for neural network approaches that require vast datasets and computational resources for training, but works well for non-neural approaches with engineered features [ 22 ]. Second, model updating (online learning, transfer learning) with user-specific data is feasible for both non-neural [ 23 , 24 , 25 ] and neural network supervised learning algorithms [ 26 , 29 ]. Rokni et al [ 26 ] trained a generic convolution neural network architecture and adapted it to specific users by retraining the classification layer while fixing the weights of the convolutional layers with excellent results.…”

Section: Introductionmentioning

confidence: 99%

“…Second, model updating (online learning, transfer learning) with user-specific data is feasible for both non-neural [ 23 , 24 , 25 ] and neural network supervised learning algorithms [ 26 , 29 ]. Rokni et al [ 26 ] trained a generic convolution neural network architecture and adapted it to specific users by retraining the classification layer while fixing the weights of the convolutional layers with excellent results. A third scheme involves using classifier ensembles [ 27 , 28 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Personalized Activity Recognition with Deep Triplet Embeddings

Burns

Whyne

2022

Sensors

View full text Add to dashboard Cite

A significant challenge for a supervised learning approach to inertial human activity recognition is the heterogeneity of data generated by individual users, resulting in very poor performance for some subjects. We present an approach to personalized activity recognition based on deep feature representation derived from a convolutional neural network (CNN). We experiment with both categorical cross-entropy loss and triplet loss for training, and describe a novel loss function based on subject triplets. We evaluate these methods on three publicly available inertial human activity recognition datasets (MHEALTH, WISDM, and SPAR) comparing classification accuracy, out-of-distribution activity detection, and generalization to new activity classes. The proposed triplet algorithm achieved an average 96.7% classification accuracy across tested datasets versus the 87.5% achieved by the baseline CNN algorithm. We demonstrate that personalized algorithms, and, in particular, the proposed novel triplet loss algorithms, are more robust to inter-subject variability and thus exhibit better performance on classification and out-of-distribution detection tasks.

show abstract

Personalized Human Activity Recognition Using Convolutional Neural Networks

Cited by 44 publications

References 5 publications

Hierarchical Dual Attention-Based Recurrent Neural Networks for Individual and Group Activity Recognition in Games

Hierarchical Dual Attention-Based Recurrent Neural Networks for Individual and Group Activity Recognition in Games

Federated Meta-Learning with Attention for Diversity-Aware Human Activity Recognition

Personalized Activity Recognition with Deep Triplet Embeddings

Contact Info

Product

Resources

About