Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation

Abstract

In socially assistive robotics, human activity recognition plays a central role when the adaptation of the robot behavior to the human one is required. In this paper, we present an activity recognition approach for activities of daily living based on deep learning and skeleton data. In the literature, ad hoc features extraction/selection algorithms with supervised classification methods have been deployed, reaching an excellent classification performance. Here, we propose a deep learning approach, combining CNN and LSTM, that exploits both the learning of spatial dependencies correlating the limbs in a skeleton 3D grid representation and the learning of temporal dependencies from instances with a periodic pattern that works on raw data and so without requiring an explicit feature extraction process. These models are proposed for real-time activity recognition, and they are tested on the CAD-60 dataset. Results show that the proposed model behaves better than an LSTM model thanks to the automatic features extraction of the limbs’ correlation. “New Person” results show that the CNN-LSTM model achieves $$95.4\%$$ 95.4 % of precision and $$94.4\%$$ 94.4 % of recall, while the “Have Seen” results are $$96.1\%$$ 96.1 % of precision and $$94.7\%$$ 94.7 % of recall.

ConvAE-LSTM: Convolutional Autoencoder Long Short-Term Memory Network for Smartphone-Based Human Activity Recognition

“…Mutegeki et al [76] used the UCI smartphone sensor dataset to identify human activities using a CNN-LSTM architecture. Ercolano and Rossi [77] proposed a CNN-LSTM-based architecture using video data (the CAD-60 dataset) for HAR. In all the aforementioned works, researchers used combinations of CNNs and LSTM to extract spatial and temporal features.…”

Personalized home-care support for the elderly: a field experience with a social robot at home

“…It should be noted that since no environmental sensors are used to detect the user position or activity, Find, Look and Approach User are microservices provided by the robot. In the same way, the Activity Recognition microservice is realized by using the robot camera once the user is recognized by the robot (Ercolano and Rossi 2021). Moreover, there are monitoring services whose results are registered in the user profile database as Daily Observations to take into account dynamic events impacting the user's state.…”

“…These ethical issues pose challenges for the evaluation of results regarding the perception of the received assistance from the user, and the possibility of correctly evaluating the performance of some developed services in the wild. For example, while activity recognition was tested on laboratory data (Ercolano and Rossi 2021), no video recording, as well as skeleton tracking, was included as means of validation during the experimentation. How to properly evaluate these functionalities in an ecologically relevant environment is an open issue.…”

Flexible Multi-Objective Particle Swarm Optimization Clustering with Game Theory to Address Human Activity Recognition Fully Unsupervised

“…Most methods introduced for HAR are supervised focusing in Fig 1 part (d) and (e), training from lablled data or data with known. They described the relation between skeleton joints in groups activity recognition [27], used deep reinforcement learning and formulate joints as a Markov model to select informative ones [28], continuously learned from skeleton activity using a brain-inspired elastic network [29], and used a hybrid CNN-LSTM to extract spatial and temporal features [30]. The performance of these methods was strongly dependent on training data labeled with ground truth.…”