Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors

Abstract

Worldwide demographic projections point to a progressively older population. This fact has fostered research on Ambient Assisted Living, which includes developments on smart homes and social robots. To endow such environments with truly autonomous behaviours, algorithms must extract semantically meaningful information from whichever sensor data is available. Human activity recognition is one of the most active fields of research within this context. Proposed approaches vary according to the input modality and the environments considered. Different from others, this paper addresses the problem of recognising heterogeneous activities of daily living centred in home environments considering simultaneously data from videos, wearable IMUs and ambient sensors. For this, two contributions are presented. The first is the creation of the Heriot-Watt University/University of Sao Paulo (HWU-USP) activities dataset, which was recorded at the Robotic Assisted Living Testbed at Heriot-Watt University. This dataset differs from other multimodal datasets due to the fact that it consists of daily living activities with either periodical patterns or long-term dependencies, which are captured in a very rich and heterogeneous sensing environment. In particular, this dataset combines data from a humanoid robot’s RGBD (RGB + depth) camera, with inertial sensors from wearable devices, and ambient sensors from a smart home. The second contribution is the proposal of a Deep Learning (DL) framework, which provides multimodal activity recognition based on videos, inertial sensors and ambient sensors from the smart home, on their own or fused to each other. The classification DL framework has also validated on our dataset and on the University of Texas at Dallas Multimodal Human Activities Dataset (UTD-MHAD), a widely used benchmark for activity recognition based on videos and inertial sensors, providing a comparative analysis between the results on the two datasets considered. Results demonstrate that the introduction of data from ambient sensors expressively improved the accuracy results.

Characterizing Smart Environments as Interactive and Collective Platforms: A Review of the Key Behaviors of Responsive Architecture

“…This type of IB can impact on individuals, communities and cultures linked through myriad sensory devices in a dynamic environment, developing “architectural responsiveness” [ 158 ]. Such IB relies on ambient recognition and intelligence to collect human activities and environmental data [ 159 ], presenting an augmented space, or so-called “ sensponsive architecture ” [ 43 ]. In addition, smart environments do not just enable individuals’ interactions with their surroundings, but also support social CBs with the physical environment.…”

Multimodal Approaches for Indoor Localization for Ambient Assisted Living in Smart Homes

“…Next, we review some of the recent AAL-based technologies that have focused on activity recognition and analysis in Smart Homes. An activity recognition framework was proposed by Ranieri et al [46] for AAL of elderly in Smart Homes. The framework focused on studying various parameters of user interaction data from videos, wearable sensors, and ambient sensors to analyze activities.…”

Daily Living Activity Recognition In-The-Wild: Modeling and Inferring Activity-Aware Human Contexts

“…In addition, smartphone-based sensors are not sufficiently adept at tracking the user's activities relating to hand movements or gestures, for example, eating, smoking, drinking, tooth brushing, etc. Henceforth, the researchers emphasized using heterogeneous and multimodal sensors for AR [29][30][31][32], which has enhanced AR system performance in most cases. Most of the existing AR methods are generally developed and implemented in restricted environments and settings to learn and identify a particular set of human activities [33].…”