Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions

Abstract

Computer vision (CV) has achieved great success in interpreting semantic meanings from images, yet CV algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. One of this tasks is in-bed human pose estimation, which has significant values in many healthcare applications. In-bed pose monitoring in natural settings could involve complete darkness or full occlusion. Furthermore, the lack of publicly available in-bed pose datasets hinders the use of many successful pose estimation algorithms for this task. In this paper, we introduce our Simultaneously-collected multimodal Lying Pose (SLP) dataset, which includes in-bed pose images from 109 participants captured using multiple imaging modalities including RGB, long wave infrared, depth, and pressure map. We also present a physical hyper parameter tuning strategy for ground truth pose label generation under extreme conditions such as lights off and being fully covered by a sheet/blanket. SLP design is compatible with the mainstream human pose datasets, therefore, the state-of-the-art 2D pose estimation models can be trained effectively with SLP data with promising performance as high as 95% at PCKh@0.5 on a single modality. The pose estimation performance can be further improved by including additional modalities through collaboration.

Pressure Eye: In-bed Contact Pressure Estimation via Contact-less Imaging

“…• Evaluating PEye performance on the largest-ever in-bed multimodal human pose dataset with >100 human subjects with nearly 15,000 pose images called Simultaneously-collected multimodal Lying Pose (SLP), including RGB, LWIR, and contact pressure map, which is publicly available in our webpage Liu et al (2020).…”

“…To evaluate PEye approach effectiveness in generating pressure data from vision signals, in our previous work Liu et al (2020), we formed and publicly released the Simultaneously-collected multimodal Lying Pose (SLP) dataset, where RGB, LWIR and PM signals are collected using a Logitech webcam, a FLIR IR camera, and a Tekscan pressure sensing mat, respectively. We collected data from 102 subjects that were instructed to lie down on a twin-size bed and take random poses in natural ways.…”

BodyPressure -- Inferring Body Pose and Contact Pressure from a Depth Image

“…To address the challenge of heavy occlusion when inferring human pose at rest, prior work has required multiple modalities as input, including RGB, depth, thermal, and pressure imagery [2], [3]. Our deep network, BodyPres-sureWnet, employs only a depth camera, but when trained on enough data-over 100, 000 samples-it substantially outperforms prior work.…”

“…Deep models for human pose estimation are highly sensitive to the pose distribution in the training data. To generate the synthetic bodies at rest, we initialize the simulator with poses that are close to real poses in the Simultaneouslycollected Lying Pose (SLP) dataset [2]. As the SLP dataset only has 2D pose annotations, we present an annotation method to fit the 3D Skinned Multi-Person Linear (SMPL) body model [7] to the real data.…”

“…Inferring the pressure distribution underneath a person from a depth image involves three main steps: (1) inferring human pose as a mesh model, (2) inferring the contact pressure on the top surface of the bed, and (3) projecting the pressure from the bed surface onto the body mesh. We introduce a network architecture to address this challenge, which uses a convolutional neural network (CNN) encoder to estimate the parameters for a SMPL parametric human model [7].…”