Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Zhou, Haiyang; Zhao, Yixin; Liu, Yanzhong; Lu, Sichao; An, Xiang; Liu, Qiang

doi:10.3390/s23104750

Cited by 11 publications

(2 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They provide a wealth of information about the temperature of the observed object, which is more resilient to variations in illumination or background clutter compared to traditional cameras. Thermal imaging cameras have been successfully utilized in various computer vision tasks, including pedestrian detection [1], behavior recognition [2], military surveillance [3], autonomous navigation in GPS-denied environments [4], fire detection [5], heat leakage detection [6], gas leak detection [7], and more. However, thermal sensors capable of capturing high-resolution images tend to be expensive.…”

Section: Introductionmentioning

confidence: 99%

Thermal image enhancement by artificial multiscale-exposure image fusion

Voronin,

Gapon,

Zhadanova

et al. 2024

Multimodal Image Exploitation and Learning 2024

View full text Add to dashboard Cite

Infrared and thermal images have been used widely in different security applications. One of the drawbacks of such images is low contrast and noisy images, which should be enhanced. We present a new image enhancement algorithm based on block-rooting processing with artificial multi-scale-exposure image fusion. The proposed block-based multi-scale enhancement method is based on a 3-D block-rooting transform domain technique comprised: finding similar blocks in the image by block-matching; block-grouping for different block sizes; applying 3-D block-matching image enhancement; decomposition of the weight map and multi-scale enhanced images into the Gaussian and Laplacian pyramids; fusion by multiplying multi-scale images and weights. A new stage is proposed to obtain a local-global estimate of high-contrast images, also used in the general artificial fusion model. Some presented experimental results illustrate the performance of the proposed method on the thermal image dataset compared with the traditional methods.

show abstract

Section: Introductionmentioning

confidence: 99%

Thermal image enhancement by artificial multiscale-exposure image fusion

Voronin,

Gapon,

Zhadanova

et al. 2024

Multimodal Image Exploitation and Learning 2024

View full text Add to dashboard Cite

show abstract

“…For instance, Namazi et al developed a method to detect and track continuous objects surrounding vehicles, locate objects in front of the vehicle, and mitigate occlusion issues to provide more precise readings [4]. Similarly, multiple systems, Zhou et al [5] and Yuan et al [6] proposed systems to recognize human activities in low-light indoor conditions by using multiple sensor devices.…”

Section: Introductionmentioning

confidence: 99%

Detection of Sensors Used for Adversarial Examples against Machine Learning Models

Kurniawan,

Ohsita,

Maisuria

et al. 2023

Preprint

View full text Add to dashboard Cite

Machine learning (ML) systems that use sensors obtain observations from sensors and use them to recognize and interpret the current situation. These systems are susceptible to sensor-based adversarial example attacks (AEs); if some sensors are vulnerable and can be compromised by an attacker, the attacker can change the output of the system by changing the values of the sensors. The detection of compromised sensors is important to defend the system against sensor-based AEs, because we can check the sensors and replace them by detecting the sensors used by the attacker. In this paper, we propose a method to detect the sensors used in sensor-based AEs by utilizing the features of the attack that cannot be avoided. In this method, we introduced a model called the feature-removable model (FRM), which allows us to select the features used as inputs into the model. Our method detects the sensors used in sensor-based AEs by finding the inconsistencies between the outputs of the FRM obtained by changing the selected features. We evaluated our method using a human activity recognition model with sensors attached to the user’s chest, wrist, and ankle. We demonstrate that our method can accurately detect sensors used by the attacker and achieves an average Recall of Detection of 0.92, and the average Precision of Detection is 0.72.

show abstract

Enhanced Aiot Multi‐Modal Fusion for Human Activity Recognition in Ambient Assisted Living Environment

Patel,

Jhaveri,

Patel

et al. 2024

Softw Pract Exp

View full text Add to dashboard Cite

MethodologyHuman activity recognition (HAR) has emerged as a fundamental capability in various disciplines, including ambient assisted living, healthcare, human‐computer interaction, etc. This study proposes a novel approach for activity recognition by integrating IoT technologies with Artificial Intelligence and Edge Computing. This work presents a fusion HAR approach that combines data readings from wearable sensors such as accelerometer and gyroscope sensors and Images captured by vision‐based sensors such as cameras incorporating the capabilities of Long Short‐Term Memory (LSTM) and Convolutional Neural Network (CNN) models. The aim of fusing these models is to capture and extract the temporal and spatial information, improving the accuracy and resilience of activity identification systems. The work uses the CNN model to find spatial features from the images that represent the contextual information of the activities and the LSTM model for processing sequential accelerometer and gyroscope sensor data to extract the temporal dynamics from the human activities.ResultsThe performance of our fusion approach is evaluated through different experiments using varying parameters and applies the best‐suited parameters for our model. The results demonstrate that the fusion of LSTM and CNN models outperforms standalone models and traditional fusion methods, achieving an accuracy of 98%, which is almost 9% higher than standalone models.ConclusionThe fusion of LSTM and CNN models enables the integration of complementary information from both data sources, leading to improved performance. The computation tasks are performed at the local edge device resulting to enhanced privacy and reduced latency. Our approach greatly impacts real‐world applications where accurate and reliable HAR systems are essential for enhancing human‐machine interaction and monitoring human activities in various domains.

show abstract

Multi-Sensor Data Fusion and CNN-LSTM Model for Human Activity Recognition System

Cited by 11 publications

References 33 publications

Thermal image enhancement by artificial multiscale-exposure image fusion

Thermal image enhancement by artificial multiscale-exposure image fusion

Detection of Sensors Used for Adversarial Examples against Machine Learning Models

Enhanced Aiot Multi‐Modal Fusion for Human Activity Recognition in Ambient Assisted Living Environment

Contact Info

Product

Resources

About