Real-time multimodal ADL recognition using convolution neural networks

Madhuranga, Danushka; Madushan, Rivindu; Siriwardane, Chathuranga; Gunasekera, Kutila

doi:10.1007/s00371-020-01864-y

Cited by 20 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, Couprie et al [214] proposed a bimodal CNN architecture for multiscale feature extraction from RGB-D datasets, which are taken as four-channel frames (blue, green, red, and depth). Similarly, Madhuranga et al [215] used CNN models for video recognition purposes by extracting silhouettes from depth sequences and then fusing the depth information with audio descriptions for activity of daily living (ADL) recognition. Zhang et al [217] proposed to use multicolumn CNNs to extract visual features from the face and eye images for the gaze point estimation problem.…”

Section: Convolutional Neural Network Basedmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

The research progress in multimodal learning has grown rapidly over the last decade in several areas, especially in computer vision. The growing potential of multimodal data streams and deep learning algorithms has contributed to the increasing universality of deep multimodal learning. This involves the development of models capable of processing and analyzing the multimodal information uniformly. Unstructured real-world data can inherently take many forms, also known as modalities, often including visual and textual content. Extracting relevant patterns from this kind of data is still a motivating goal for researchers in deep learning. In this paper, we seek to improve the understanding of key concepts and algorithms of deep multimodal learning for the computer vision community by exploring how to generate deep models that consider the integration and combination of heterogeneous visual cues across sensory modalities. In particular, we summarize six perspectives from the current literature on deep multimodal learning, namely: multimodal data representation, multimodal fusion (i.e., both traditional and deep learning-based schemes), multitask learning, multimodal alignment, multimodal transfer learning, and zero-shot learning. We also survey current multimodal applications and present a collection of benchmark datasets for solving problems in various vision domains. Finally, we highlight the limitations and challenges of deep multimodal learning and provide insights and directions for future research.

show abstract

Section: Convolutional Neural Network Basedmentioning

confidence: 99%

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The proposed model has achieved an accuracy of 90.8%. Madhuranga et al [14] employed CNN in-depth images that were previously used for RGB video for silhouette segmentation. They have developed the model for real-time recognition, which is capable of recognizing ADL on commodity computers.…”

Section: Related Workmentioning

confidence: 99%

Self-Care Assessment for Daily Living Using Machine Learning Mechanism

Batool¹,

Ghadi²,

Alsuhibany³

et al. 2022

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Nowadays, activities of daily living (ADL) recognition system has been considered an important field of computer vision. Wearable and optical sensors are widely used to assess the daily living activities in healthy people and people with certain disorders. Although conventional ADL utilizes RGB optical sensors but an RGB-D camera with features of identifying depth (distance information) and visual cues has greatly enhanced the performance of activity recognition. In this paper, an RGB-D-based ADL recognition system has been presented. Initially, human silhouette has been extracted from the noisy background of RGB and depth images to track human movement in a scene. Based on these silhouettes, full body features and point based features have been extracted which are further optimized with probability based incremental learning (PBIL) algorithm. Finally, random forest classifier has been used to classify activities into different categories. The n-fold crossvalidation scheme has been used to measure the viability of the proposed model on the RGBD-AC benchmark dataset and has achieved an accuracy of 92.71% over other state-of-the-art methodologies.

show abstract

“…HAR approaches are divided into vision-based and sensor-based. Vision-based HAR systems (Madhuranga et al, 2021) rely on visual sensing technologies (cameras and CCTV) to record human activities. Even though these systems have achieved decent performance in recognizing various activities, they have some drawbacks, like being too expensive, limited area coverage, privacy concerns, and so forth.…”

Section: Introductionmentioning

confidence: 99%

An optimized deep learning model for human activity recognition using inertial measurement units

Challa,

Kumar,

Semwal

et al. 2023

Expert Systems

View full text Add to dashboard Cite

Human activity recognition (HAR) has recently gained popularity due to its applications in healthcare, surveillance, human‐robot interaction, and various other fields. Deep learning (DL)‐based models have been successfully applied to the raw data captured through inertial measurement unit (IMU) sensors to recognize multiple human activities. Despite the success of DL‐based models in human activity recognition, feature extraction remains challenging due to class imbalance and noisy data. Additionally, selecting optimal hyperparameter values for DL models is essential since they affect model performance. The hyperparameter values of some of the existing DL‐based HAR models are chosen randomly or through the trial‐and‐error method. The random selection of these significant hyperparameters may be suitable for some applications, but sometimes it may worsen the model's performance in others. Hence, to address the above‐mentioned issues, this research aims to develop an optimized DL model capable of recognizing various human activities captured through IMU sensors. The proposed DL‐based HAR model combines convolutional neural network (CNN) layers and bidirectional long short‐term memory (Bi‐LSTM) units to simultaneously extract spatial and temporal sequence features from raw sensor data. The Rao‐3 metaheuristic optimization algorithm has been adopted to identify the ideal hyperparameter values for the proposed DL model in order to enhance its recognition performance. The proposed DL model's performance is validated on PAMAP2, UCI‐HAR, and MHEALTH datasets and achieved 94.91%, 97.16%, and 99.25% accuracies, respectively. The results reveal that the proposed DL model performs better than the existing state‐of‐the‐art (SoTA) models.

show abstract

Real-time multimodal ADL recognition using convolution neural networks

Cited by 20 publications

References 40 publications

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Self-Care Assessment for Daily Living Using Machine Learning Mechanism

An optimized deep learning model for human activity recognition using inertial measurement units

Contact Info

Product

Resources

About