LSTM-CNN Architecture for Human Activity Recognition

Abstract

In the past years, traditional pattern recognition methods have made great progress. However, these methods rely heavily on manual feature extraction, which may hinder the generalization model performance. With the increasing popularity and success of deep learning methods, using these techniques to recognize human actions in mobile and wearable computing scenarios has attracted widespread attention. In this paper, a deep neural network that combines convolutional layers with long short-term memory (LSTM) was proposed. This model could extract activity features automatically and classify them with a few model parameters. LSTM is a variant of the recurrent neural network (RNN), which is more suitable for processing temporal sequences. In the proposed architecture, the raw data collected by mobile sensors was fed into a two-layer LSTM followed by convolutional layers. In addition, a global average pooling layer (GAP) was applied to replace the fully connected layer after convolution for reducing model parameters. Moreover, a batch normalization layer (BN) was added after the GAP layer to speed up the convergence, and obvious results were achieved. The model performance was evaluated on three public datasets (UCI, WISDM, and OPPORTUNITY). Finally, the overall accuracy of the model in the UCI-HAR dataset is 95.78%, in the WISDM dataset is 95.85%, and in the OPPORTUNITY dataset is 92.63%. The results show that the proposed model has higher robustness and better activity detection capability than some of the reported results. It can not only adaptively extract activity features, but also has fewer parameters and higher accuracy. INDEX TERMS Human activity recognition, convolution, long short-term memory, mobile sensors.

Human Daily Activity Recognition Performed Using Wearable Inertial Sensors Combined With Deep Learning Algorithms

“…Reference [20] used a threelayer LSTM model. Although reference [28] has an accuracy similar to ours, like references [33] and [34] it did not use only CNN but also LSTM, and thus the complexity of the algorithm was increased. This study used only CNN and could obtain similar or even superior accuracy.…”

“…After 10 repetitions, the average accuracy of the open database was 95.08%, and the mean accuracy of the database we recorded was 87.88%. [20] 93.70% LSTM-CNN [28] 95.78% Bidir-LSTM [31] 93.79% EHARS [32] 93.92% CNN-LSTM [33] 92.13% CNN-LSTM [34] 93.40% Ours 95.99%…”

“…Zebin et al proposed a deep CNN model to classify five daily activities: walking, walking upstairs, walking downstairs, sitting for long periods of time and sleeping; the raw data from the accelerometer and the gyroscope of the wearable device served as the input, and the accuracy was 96.4% [27]. Xia et al proposed a deep neural network combining the convolutional layer and LSTM; the model automatically performed feature extraction and classification without requiring manual feature extraction in conventional recognition approaches, and the accuracies for three open datasets (UCI, WISDM, and OPPORTUNITY) were 95.78%, 95.85%, and 92.63%, respectively [28]. Kańtoch indicated the correlation of sedentary lifestyle with increased pathogenic risk; seven healthy volunteers performed routine activities-sitting, walking, standing and squats-and the overall accuracy reached 82% [29].…”

Biometric User Identification Based on Human Activity Recognition Using Wearable Sensors: An Experiment Using Deep Learning Models

“…In particular, it is interesting to observe that both activity dataset (UCI HAR and USC HAD) are well classified by the CNN-LSTM deep learning model. However, these experimental results are not state-of-the-art results [38,39].…”

Combined Deep CNN–LSTM Network-based Multitasking Learning Architecture for Noninvasive Continuous Blood Pressure Estimation using Difference in ECG-PPG Features

“…, we developed an NIBP algorithm using a combined deep CNN-LSTM network-based multitasking learning architecture. The combined deep CNN-LSTM model was constructed based on the LSTM-CNN model of Xia et al[14] to extract morphological and temporal features from the signal difference between ECG and PPG.…”