Uncovering Human Multimodal Activity Recognition with a Deep Learning Approach

Ranieri, Caetano Mazzoni; Vargas, Patrícia A.; Romero, Roseli Aparecida Francelin

doi:10.1109/ijcnn48605.2020.9207255

Cited by 12 publications

(8 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As for the classifiers, we built on DL architectures for data from video and inertial sensors, presented on our previous work [ 36 ]. The most relevant contributions of this paper are the models trained not only on data from those modalities, but also considering ambient sensors from the smart home.…”

Section: Methodsmentioning

confidence: 99%

“…For feature extraction and classification of activities recorded with videos and inertial data, we have proposed different DL architectures and compared the resulting models and their accuracies on a previous work [ 36 ]. Based on the results obtained in this previous paper, we chose the CNN and LSTM models as basis for our experiments.…”

Section: Methodsmentioning

confidence: 99%

“…Although different possibilities for fusion of multimodal data using DL methods have been proposed, especially regarding to different inputs from multiple video streams [ 95 ], the most popular approach for dealing with heterogeneous data is to process each modality separately and fuse the obtained scores at a later stage [ 35 , 36 ], which we refer as late fusion . This was the approach adopted on all experiments performed in this paper.…”

Section: Human Activity Recognitionmentioning

confidence: 99%

“…The fusion between IMU data and ambient sensors was performed internally as part of one of the DL architectures presented, after both modalities were temporally aligned in a preprocessing stage, an approach that we are proposing as part of this work. To perform fusion between the video-based models and the models that processed IMU and ambient sensors’ data, the output vectors were combined with the outputs of the other modalities, also an approach commonly adopted in related research [ 35 , 36 ].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors

Ranieri

MacLeod

Dragone

et al. 2021

Sensors

View full text Add to dashboard Cite

Worldwide demographic projections point to a progressively older population. This fact has fostered research on Ambient Assisted Living, which includes developments on smart homes and social robots. To endow such environments with truly autonomous behaviours, algorithms must extract semantically meaningful information from whichever sensor data is available. Human activity recognition is one of the most active fields of research within this context. Proposed approaches vary according to the input modality and the environments considered. Different from others, this paper addresses the problem of recognising heterogeneous activities of daily living centred in home environments considering simultaneously data from videos, wearable IMUs and ambient sensors. For this, two contributions are presented. The first is the creation of the Heriot-Watt University/University of Sao Paulo (HWU-USP) activities dataset, which was recorded at the Robotic Assisted Living Testbed at Heriot-Watt University. This dataset differs from other multimodal datasets due to the fact that it consists of daily living activities with either periodical patterns or long-term dependencies, which are captured in a very rich and heterogeneous sensing environment. In particular, this dataset combines data from a humanoid robot’s RGBD (RGB + depth) camera, with inertial sensors from wearable devices, and ambient sensors from a smart home. The second contribution is the proposal of a Deep Learning (DL) framework, which provides multimodal activity recognition based on videos, inertial sensors and ambient sensors from the smart home, on their own or fused to each other. The classification DL framework has also validated on our dataset and on the University of Texas at Dallas Multimodal Human Activities Dataset (UTD-MHAD), a widely used benchmark for activity recognition based on videos and inertial sensors, providing a comparative analysis between the results on the two datasets considered. Results demonstrate that the introduction of data from ambient sensors expressively improved the accuracy results.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Human Activity Recognitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors

Ranieri

MacLeod

Dragone

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…To provide a wider range of possibilities, robots may act symbiotically with other pervasive devices, such as wearable technologies or ambient sensors in intelligent environments, which may provide additional capabilities for sensing and acting based on application-specific components [2]. When synchronised data from different sensors are available, activity recognition techniques may rely on multiple sensor modalities to provide more accurate results, giving rise to techniques for multimodal activity recognition [28,17,43].…”

Section: Introductionmentioning

confidence: 99%

A Neurorobotics Approach to Behaviour Selection based on Human Activity Recognition

Ranieri,

Moioli,

Vargas

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Behaviour selection has been an active research topic for robotics, in particular in the field of human-robot interaction. For a robot to interact effectively and autonomously with humans, the coupling between techniques for human activity recognition, based on sensing information, and robot behaviour selection, based on decision-making mechanisms, is of paramount importance. However, most approaches to date consist of deterministic associations between the recognised activities and the robot behaviours, neglecting the uncertainty inherent to sequential predictions in real-time applications. In this paper, we address this gap by presenting a neurorobotics approach based on computational models that resemble neurophysiological aspects of living beings. This neurorobotics approach was compared to a non-bioinspired, heuristics-based approach. To evaluate both approaches, a robot simulation is developed, in which a mobile robot has to accomplish tasks according to the activity being performed by the inhabitant of an intelligent home. The outcomes of each approach were evaluated according to the number of correct outcomes provided by the robot. Results revealed that the neurorobotics approach is advantageous, especially considering the computational models based on more complex animals.

show abstract