Person Re-ID by Fusion of Video Silhouettes and Wearable Signals for Home Monitoring Applications

Abstract

The use of visual sensors for monitoring people in their living environments is critical in processing more accurate health measurements, but their use is undermined by the issue of privacy. Silhouettes, generated from RGB video, can help towards alleviating the issue of privacy to some considerable degree. However, the use of silhouettes would make it rather complex to discriminate between different subjects, preventing a subject-tailored analysis of the data within a free-living, multi-occupancy home. This limitation can be overcome with a strategic fusion of sensors that involves wearable accelerometer devices, which can be used in conjunction with the silhouette video data, to match video clips to a specific patient being monitored. The proposed method simultaneously solves the problem of Person ReID using silhouettes and enables home monitoring systems to employ sensor fusion techniques for data analysis. We develop a multimodal deep-learning detection framework that maps short video clips and accelerations into a latent space where the Euclidean distance can be measured to match video and acceleration streams. We train our method on the SPHERE Calorie Dataset, for which we show an average area under the ROC curve of 76.3% and an assignment accuracy of 77.4%. In addition, we propose a novel triplet loss for which we demonstrate improving performances and convergence speed.

Amazon Echo Show as a Multimodal Human-to-Human Care Support Tool within Self-Isolating Older UK Households

“…Providing greater support to informal caregivers and other household residents who may interact with the device and perform actions with it, on behalf of another resident, should be considered. Investigations into this type of feature support could build on existing work in the AI domain, detecting multi-user interactions with smart devices [44,45].…”

Simulation-Driven Design of Smart Gloves for Gesture Recognition

“…Because video and IMU sensors provide heterogeneous data streams, the late feature fusion is a popular technique for combining the modalities. For example, Masullo et al [94] focused on privacy-preserving techniques for home monitoring through videos and wearable signals. User re-identification and monitoring were achieved through silhouettes extracted from videos and then merged with acceleration readings to estimate calorie expenditure for different activities.…”

Multimodal Classification of Parkinson’s Disease in Home Environments with Resiliency to Missing Modalities

“…Moreover, while vision has proved to be a powerful modality for evaluating PD, privacy issues in home settings has limited research on RGB data. To deal with this, we reduce the means for identification by taking an approach similar to [ 28 , 29 ], in which human silhouettes are extracted and RGB and depth data are discarded.…”

“…[ 33 , 34 ] use transformer-based models to discover the inherent semantic correlations between vision and language. However, a relatively small part of research in the multimodal learning literature deviates by focusing on other data types such as vision and body-worn IMU data [ 28 , 29 , 41 , 42 ], where the modalities are mainly correlated due to the body movements of the subjects. Among these works, [ 28 ] proposes a network, called CaloriNet, for fusing accelerometer and silhouette data to estimate the calorie expenditure of the subjects.…”