Depth-Camera-Based Under-Blanket Sleep Posture Classification Using Anatomical Landmark-Guided Deep Learning Model

Abstract

Emerging sleep health technologies will have an impact on monitoring patients with sleep disorders. This study proposes a new deep learning model architecture that improves the under-blanket sleep posture classification accuracy by leveraging the anatomical landmark feature through an attention strategy. The system used an integrated visible light and depth camera. Deep learning models (ResNet-34, EfficientNet B4, and ECA-Net50) were trained using depth images. We compared the models with and without an anatomical landmark coordinate input generated with an open-source pose estimation model using visible image data. We recruited 120 participants to perform seven major sleep postures, namely, the supine posture, prone postures with the head turned left and right, left- and right-sided log postures, and left- and right-sided fetal postures under four blanket conditions, including no blanket, thin, medium, and thick. A data augmentation technique was applied to the blanket conditions. The data were sliced at an 8:2 training-to-testing ratio. The results showed that ECA-Net50 produced the best classification results. Incorporating the anatomical landmark features increased the F1 score of ECA-Net50 from 87.4% to 92.2%. Our findings also suggested that the classification performances of deep learning models guided with features of anatomical landmarks were less affected by the interference of blanket conditions.

Vision Transformers (ViT) for Blanket-Penetrating Sleep Posture Recognition Using a Triple Ultra-Wideband (UWB) Radar System

“…The long-term objective of this research is to develop a comprehensive sleep surveillance system that can monitor sleep postures and behaviors. Our previous studies developed a depth camera system to monitor bed-exiting events [63,64] and to classify sleep postures in a field setting [14,15,28]. In the future, we will explore synthetic aperture radar and advanced modeling techniques, for instance, DensePose, which could estimate and map the human pixels of an RGB image to the 3D surface of the human body in real time [65,66].…”

“…The pressure intensity distribution generated by a pressure mat has been utilized to characterize sleep postural behavior and estimated sleep quality [11][12][13]. Video recordings using red-green-blue (RGB) or red-green-blue-depth (RGB-D) images can capture and facilitate observation of the sleep postures of individuals directly [14][15][16][17]. Wearable devices using actigraphy or accelerometry can measure physical activity and infer motor or behavioral activities [18].…”

“…Zhou et al [31] transformed a radar signal to image features, which were then handled by a CNN model integrated with an inception-residual module. Tam et al [14] guided a CNN-based deep learning model (ECA-Net) by generating anatomical landmarks on depth images using a pose estimator. In addition, using random forest, Lai et al [28] extracted radar features from each radar bin range for posture recognition.…”

Assessing Non-Specific Neck Pain through Pose Estimation from Images Based on Ensemble Learning

“…In addition, environmental and ergonomic factors such as the depth of the , the height of the table, and the height of the chair play crucial roles in poor posture. Many studies focused on associations among neck pain, poor posture, spinal mobility, and exercise [3][4][5][6][7], and some methods of analyzing posture have been implemented utilizing advanced technology, such as depth cameras [8] or radar systems [9]. Although medical treatments and physiotherapy can effectively relieve neck pain, prevention through adequate patient education is essential for managing neck pain.…”

Transformer Models and Convolutional Networks with Different Activation Functions for Swallow Classification Using Depth Video Data

“…Recently, noncontact optical-based approaches using infrared depth cameras have emerged and been adopted for different mobile health applications [30][31][32][33]. Specific to dysphagia, An et al [34] developed a liquid viscosity estimation model using the builtin camera of the smartphone with a convolutional neural network (CNN).…”