Deep CNN-LSTM With Self-Attention Model for Human Activity Recognition Using Wearable Sensor

Khatun, Mst. Alema; Yousuf, Mohammad Abu; Ahmed, Shamsuddin; Uddin, Md. Zia; Alyami, Salem A.; Al-Ashhab, Samer; Akhdar, Hanan; Khan, Asaduzzaman; Azad, A. K. M.; Moni, Mohammad Ali

doi:10.1109/jtehm.2022.3177710

Cited by 123 publications

(54 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data interpolation. During the practical data collection using smartphone sensors, some data points in the acquired dataset are lost due to malfunctioning; such data points are typically replaced by 0, NaN, or none [ 33 ]. To fill in the missing values, the data interpolation technique was developed, in which the new data point is estimated based on the known information.…”

Section: Methodsmentioning

confidence: 99%

Indoor Spatiotemporal Contact Analytics Using Landmark-Aided Pedestrian Dead Reckoning on Smartphones

Gao

Konomi

2022

Sensors

View full text Add to dashboard Cite

Due to the prevalence of COVID-19, providing safe environments and reducing the risks of virus exposure play pivotal roles in our daily lives. Contact tracing is a well-established and widely-used approach to track and suppress the spread of viruses. Most digital contact tracing systems can detect direct face-to-face contact based on estimated proximity, without quantifying the exposed virus concentration. In particular, they rarely allow for quantitative analysis of indirect environmental exposure due to virus survival time in the air and constant airborne transmission. In this work, we propose an indoor spatiotemporal contact awareness framework (iSTCA), which explicitly considers the self-containing quantitative contact analytics approach with spatiotemporal information to provide accurate awareness of the virus quanta concentration in different origins at various times. Smartphone-based pedestrian dead reckoning (PDR) is employed to precisely detect the locations and trajectories for distance estimation and time assessment without the need to deploy extra infrastructure. The PDR technique we employ calibrates the accumulative error by identifying spatial landmarks automatically. We utilized a custom deep learning model composed of bidirectional long short-term memory (Bi-LSTM) and multi-head convolutional neural networks (CNNs) for extracting the local correlation and long-term dependency to recognize landmarks. By considering the spatial distance and time difference in an integrated manner, we can quantify the virus quanta concentration of the entire indoor environment at any time with all contributed virus particles. We conducted an extensive experiment based on practical scenarios to evaluate the performance of the proposed system, showing that the average positioning error is reduced to less than 0.7 with high confidence and demonstrating the validity of our system for the virus quanta concentration quantification involving virus movement in a complex indoor environment.

show abstract

Section: Methodsmentioning

confidence: 99%

Indoor Spatiotemporal Contact Analytics Using Landmark-Aided Pedestrian Dead Reckoning on Smartphones

Gao

Konomi

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Incorporating deep learning networks and combining them has shown the potential for state-of-the-art performance. For example, Khatun et al [ 41 ] proposed a DCNN-LSTM classifier with self-attention model, which was capable of attaining an accuracy of 99.93% for human activity recognition purposes. In another study for classifying MRI brain tumor, authors implemented CNN with PCA in the feature extraction step and fed these features to different machine learning classification algorithms, which yielded a remarkable 99.76% accuracy [ 42 ].…”

Section: Related Workmentioning

confidence: 99%

Newborn Cry-Based Diagnostic System to Distinguish between Sepsis and Respiratory Distress Syndrome Using Combined Acoustic Features

2022

View full text Add to dashboard Cite

Crying is the only means of communication for a newborn baby with its surrounding environment, but it also provides significant information about the newborn’s health, emotions, and needs. The cries of newborn babies have long been known as a biomarker for the diagnosis of pathologies. However, to the best of our knowledge, exploring the discrimination of two pathology groups by means of cry signals is unprecedented. Therefore, this study aimed to identify septic newborns with Neonatal Respiratory Distress Syndrome (RDS) by employing the Machine Learning (ML) methods of Multilayer Perceptron (MLP) and Support Vector Machine (SVM). Furthermore, the cry signal was analyzed from the following two different perspectives: 1) the musical perspective by studying the spectral feature set of Harmonic Ratio (HR), and 2) the speech processing perspective using the short-term feature set of Gammatone Frequency Cepstral Coefficients (GFCCs). In order to assess the role of employing features from both short-term and spectral modalities in distinguishing the two pathology groups, they were fused in one feature set named the combined features. The hyperparameters (HPs) of the implemented ML approaches were fine-tuned to fit each experiment. Finally, by normalizing and fusing the features originating from the two modalities, the overall performance of the proposed design was improved across all evaluation measures, achieving accuracies of 92.49% and 95.3% by the MLP and SVM classifiers, respectively. The MLP classifier was outperformed in terms of all evaluation measures presented in this study, except for the Area Under Curve of Receiver Operator Characteristics (AUC-ROC), which signifies the ability of the proposed design in class separation. The achieved results highlighted the role of combining features from different levels and modalities for a more powerful analysis of the cry signals, as well as including a neural network (NN)-based classifier. Consequently, attaining a 95.3% accuracy for the separation of two entangled pathology groups of RDS and sepsis elucidated the promising potential for further studies with larger datasets and more pathology groups.

show abstract

“…Khatun et al. [ 39 ] proposed a combined deep learning model (CNN-LSTM) with a self-attention mechanism for HAR smartphone applications. They assessed the quality of the combined model using two benchmark datasets, UCI-HAR and MHEALTH.…”

Section: Related Workmentioning

confidence: 99%

The Applications of Metaheuristics for Human Activity Recognition and Fall Detection Using Wearable Sensors: A Comprehensive Analysis

et al. 2022

View full text Add to dashboard Cite

In this paper, we study the applications of metaheuristics (MH) optimization algorithms in human activity recognition (HAR) and fall detection based on sensor data. It is known that MH algorithms have been utilized in complex engineering and optimization problems, including feature selection (FS). Thus, in this regard, this paper used nine MH algorithms as FS methods to boost the classification accuracy of the HAR and fall detection applications. The applied MH were the Aquila optimizer (AO), arithmetic optimization algorithm (AOA), marine predators algorithm (MPA), artificial bee colony (ABC) algorithm, genetic algorithm (GA), slime mold algorithm (SMA), grey wolf optimizer (GWO), whale optimization algorithm (WOA), and particle swarm optimization algorithm (PSO). First, we applied efficient prepossessing and segmentation methods to reveal the motion patterns and reduce the time complexities. Second, we developed a light feature extraction technique using advanced deep learning approaches. The developed model was ResRNN and was composed of several building blocks from deep learning networks including convolution neural networks (CNN), residual networks, and bidirectional recurrent neural networks (BiRNN). Third, we applied the mentioned MH algorithms to select the optimal features and boost classification accuracy. Finally, the support vector machine and random forest classifiers were employed to classify each activity in the case of multi-classification and to detect fall and non-fall actions in the case of binary classification. We used seven different and complex datasets for the multi-classification case: the PAMMP2, Sis-Fall, UniMiB SHAR, OPPORTUNITY, WISDM, UCI-HAR, and KU-HAR datasets. In addition, we used the Sis-Fall dataset for the binary classification (fall detection). We compared the results of the nine MH optimization methods using different performance indicators. We concluded that MH optimization algorithms had promising performance in HAR and fall detection applications.

show abstract

Deep CNN-LSTM With Self-Attention Model for Human Activity Recognition Using Wearable Sensor

Cited by 123 publications

References 53 publications

Indoor Spatiotemporal Contact Analytics Using Landmark-Aided Pedestrian Dead Reckoning on Smartphones

Indoor Spatiotemporal Contact Analytics Using Landmark-Aided Pedestrian Dead Reckoning on Smartphones

Newborn Cry-Based Diagnostic System to Distinguish between Sepsis and Respiratory Distress Syndrome Using Combined Acoustic Features

The Applications of Metaheuristics for Human Activity Recognition and Fall Detection Using Wearable Sensors: A Comprehensive Analysis

Contact Info

Product

Resources

About