Cardio-respiratory signal extraction from video camera data for continuous non-contact vital sign monitoring using deep learning

Abstract

Non-contact vital sign monitoring enables the estimation of vital signs, such as heart rate, respiratory rate and oxygen saturation (SpO 2 ), by measuring subtle color changes on the skin surface using a video camera. For patients in a hospital ward, the main challenges in the development of continuous and robust non-contact monitoring techniques are the identification of time periods and the segmentation of skin regions of interest (ROIs) from which vital signs can be estimated. We propose a deep learning framework to tackle these challenges. Approach : This paper presents two convolutional neural network (CNN) models. The first network was designed for detecting the presence of a patient and segmenting the patient’s skin area. The second network combined the output from the first network with optical flow for identifying time periods of clinical intervention so that these periods can be excluded from the estimation of vital signs. Both networks were trained using video recordings from a clinical study involving 15 pre-term infants conducted in the high dependency area of the neonatal intensive care unit (NICU) of the John Radcliffe Hospital in Oxford, UK. Main results : Our proposed methods achieved an accuracy of 98.8% for patient detection, a mean intersection-over-union (IOU) score of 88.6% for skin segmentation and an accuracy of 94.5% for clinical intervention detection using two-fold cross validation. Our deep learning models produced accurate results and were robust to different skin tones, changes in light conditions, pose variations and different clinical interventions by medical staff and family visitors. Significance : Our approach allows cardio-respiratory signals to be continuously derived from the patient’s skin during which the patient is present and no clinical intervention is undertaken.

Reference Free Incremental Deep Learning Model Applied for Camera-Based Respiration Monitoring

“…The involved network was trained by manually annotated skin regions and reached mean absolute error (MAE) 6.9-7.5 breaths per seconds in the collected data set including motion active regions as well. The [15] and [16] demonstrate a complex monitoring system based on a multiple output convolutional neural network. The solution gives a higher level detection capability such as intervention periods.…”

Long Short-Term Memory Deep-Filter in Remote Photoplethysmography

“…With PPG or rPPG measurement, several biomedical parameters can potentially be computed: vascular occlusion, peripheral vasomotor activity, breathing rate, blood pressure by pulse transit time estimation, blood oxygen level, heart rate variability (HRV), and obviously heart rate [1]. As a consequence, the applications are numerous, including control of vital signs in the elderly and newborns, mixed reality, lie detection in criminals, physiological measurements of drivers, face anti-spoofing, and automatic skin detection, to mention a few [6,4,1]. First of all, it can be noted that the advent of non-contact measurements has opened the way to many new applications.…”

“…Readers interested in learning more about these techniques can refer to the state of the art reviews presented in [16,11,24]. End-to-end approaches based on deep learning have also been used recently [7,22,6,4,32]. One of the main advantages of these CNN-based measurements is that it allows achieving good results without the need for the designer to analyze the problem in depth [33].…”

Respiration Signal Extraction From Pulse Wave Collected by PVDF Sensor

“…Except for directly getting respiratory information from nasal airflow or by sensor fixed on chest by belt, Reference [4] extracts cardio-respiratory signal from patient's skin from video camera data for continuous non-contact vital sign monitoring. Reference [5] estimates respiration by performing the Fourier analysis on the time sequence of the optical flow vectors from a thoracoabdominal video.…”