A Multiview Multimodal System for Monitoring Patient Sleep

Abstract

Clinical observations indicate that during critical care at the hospitals, a patient's sleep positioning and motion have a significant effect on recovery rate. Unfortunately, there is no formal medical protocol to record, quantify, and analyze motion of patients. There are very few clinical studies that use manual analysis of sleep poses and motion recordings to support medical benefits of patient positioning and motion monitoring. Manual processes do not scale, are prone to human errors, and put strain on an already taxed healthcare workforce. This study introduces Multimodal, Multiview Motion Analysis and Summarization for healthcare (MASH). MASH is an autonomous system, which addresses these issues by monitoring healthcare environments and enabling the recording and analysis of patient sleep-pose patterns. MASH uses three RGB-D cameras to monitor patients in a medical Intensive Care Unit (ICU) room. The proposed algorithms estimate pose direction at different temporal resolutions and use keyframes to efficiently represent pose transition dynamics. MASH combines deep features computed from the data with a modified version of Hidden Markov Model (HMM) to flexibly model pose duration and summarize patient motion. The performance is evaluated in ideal (BC: Bright and Clear/occlusion-free) and natural (DO: Dark and Occluded) scenarios at two motion resolutions and in two environments: a mock-up and a medical ICU. The usage of deep features is evaluated and their performance compared with engineered features. Experimental results using deep features in DO scenes increases performance from 86.7% to 93.6%, while matching the classification performance of engineered features in BC scenes. The performance of MASH is compared with HMM and C3D. The overall overtime tracing and summarization error rate across all methods increased when transitioning from the mock-up to the the medical ICU data. The proposed keyframe estimation helps achieve a 78% transition classification accuracy.

A deep neural network model for multi-view human activity recognition

“…When a subject performs activity in a dynamic environment, self and inter-object occlusion may occur. Multi-view human activity recognition helps to prevent a complete loss of information when occlusion appears in a single camera by providing information from other cameras [5,11]. Previous findings have indicated that employing multiple view increased the recognition rate of human activity in a dynamic environment [6,7].…”

“…Occlusion causes information loss and failure in single-view human activity recognition [1]. Previous researchers [2][3][4][5][6][7] have attempted to resolve this issue with the use of multiple cameras providing different angles of view [1,6] and enabling 3D posture representation [8,9]. Multi-view human action recognition (MVHAR) has a wide range of applications, including in systems for surveillance [4,10] and human behavior monitoring [5,11].…”

Multi-View Multi-Modal Head-Gaze Estimation for Advanced Indoor User Interaction

“…2, many researchers have attempted to develop contactless monitoring systems for various applications. Previous methods for monitoring people can be divided into 1) vision-based [4][5][6][7][8][9][10][11][12][13], 2) wearable-based [14][15][16][17][18][19][20][21], and 3) head-gaze-based technologies [22][23][24][25][26][27][28][29][30][31]. Vision-based methods usually detect and track the pose or movements of medically vulnerable people using user images captured by cameras.…”

Insights of Artificial Intelligence to Stop Spread of COVID-19

“…They used computer vision tasks such as: face detection and recognition, facial action unit detection and expression recognition, head pose detection and recognition, sound and light level detection, and other activity detection. A research multi-view multi-modal systems for sleep monitoring of patients [37]. Sleeping position is very vital for the recovery of a patient for some diseases in ICU, so their model has concentrated this kind of detection in ICU.…”