Objectives: Camera-based vital sign estimation allows the contactless assessment of important physiological parameters. Seminal contributions were made in the 1930s, 1980s, and 2000s, and the speed of development seems ever increasing. In this suivey, we aim to overview the most recent works in this area, describe their common features as well as shortcomings, and highlight interesting “outliers”. Methods: We performed a comprehensive literature research and quantitative analysis of papers published between 2016 and 2018. Quantitative information about the number of subjects, studies with healthy volunteers vs. pathological conditions, public datasets, laboratory vs. real-world works, types of camera, usage of machine learning, and spectral properties of data was extracted. Moreover, a qualitative analysis of illumination used and recent advantages in terms of algorithmic developments was also performed. Results: Since 2016, 116 papers were published on camera-based vital sign estimation and 59% of papers presented results on 20 or fewer subjects. While the average number of participants increased from 15.7 in 2016 to 22.9 in 2018, the vast majority of papers (n=100) were on healthy subjects. Four public datasets were used in 10 publications. We found 27 papers whose application scenario could be considered a real-world use case, such as monitoring during exercise or driving. These include 16 papers that dealt with non-healthy subjects. The majority of papers (n=61) presented results based on visual, red-green-blue (RGB) information, followed by RGB combined with other parts of the electromagnetic spectrum (n=18), and thermography only (n=12), while other works (n=25) used other mono- or polychromatic non-RGB data. Surprisingly, a minority of publications (n=39) made use of consumer-grade equipment. Lighting conditions were primarily uncontrolled or ambient. While some works focused on specialized aspects such as the removal of vital sign information from video streams to protect privacy or the influence of video compression, most algorithmic developments were related to three areas: region of interest selection, tracking, or extraction of a one-dimensional signal. Seven papers used deep learning techniques, 17 papers used other machine learning approaches, and 92 made no explicit use of machine learning. Conclusion: Although some general trends and frequent shortcomings are obvious, the spectrum of publications related to camera-based vital sign estimation is broad. While many creative solutions and unique approaches exist, the lack of standardization hinders comparability of these techniques and of their performance. We believe that sharing algorithms and/ or datasets will alleviate this and would allow the application of newer techniques such as deep learning.

Infrared thermography for camera-based skin temperature measurement is increasingly used in medical practice, e.g., to detect fevers and infections, such as recently in the COVID-19 pandemic. This contactless method is a promising technology to continuously monitor the vital signs of patients in clinical environments. In this study, we investigated both skin temperature trend measurement and the extraction of respiration-related chest movements to determine the respiratory rate using low-cost hardware in combination with advanced algorithms. In addition, the frequency of medical examinations or visits to the patients was extracted. We implemented a deep learning-based algorithm for real-time vital sign extraction from thermography images. A clinical trial was conducted to record data from patients on an intensive care unit. The YOLOv4-Tiny object detector was applied to extract image regions containing vital signs (head and chest). The infrared frames were manually labeled for evaluation. Validation was performed on a hold-out test dataset of 6 patients and revealed good detector performance (0.75 intersection over union, 0.94 mean average precision). An optical flow algorithm was used to extract the respiratory rate from the chest region. The results show a mean absolute error of 2.69 bpm. We observed a computational performance of 47 fps on an NVIDIA Jetson Xavier NX module for YOLOv4-Tiny, which proves real-time capability on an embedded GPU system. In conclusion, the proposed method can perform real-time vital sign extraction on a low-cost system-on-module and may thus be a useful method for future contactless vital sign measurements.

Photoplethysmography imaging (PPGI) for non-contact monitoring of preterm infants in the neonatal intensive care unit (NICU) is a promising technology, as it could reduce medical adhesive-related skin injuries and associated complications. For practical implementations of PPGI, a region of interest has to be detected automatically in real time. As the neonates’ body proportions differ significantly from adults, existing approaches may not be used in a straightforward way, and color-based skin detection requires RGB data, thus prohibiting the use of less-intrusive near-infrared (NIR) acquisition. In this paper, we present a deep learning-based method for segmentation of neonatal video data. We augmented an existing encoder-decoder semantic segmentation method with a modified version of the ResNet-50 encoder. This reduced the computational time by a factor of 7.5, so that 30 frames per second can be processed at 960 × 576 pixels. The method was developed and optimized on publicly available databases with segmentation data from adults. For evaluation, a comprehensive dataset consisting of RGB and NIR video recordings from 29 neonates with various skin tones recorded in two NICUs in Germany and India was used. From all recordings, 643 frames were manually segmented. After pre-training the model on the public adult data, parts of the neonatal data were used for additional learning and left-out neonates are used for cross-validated evaluation. On the RGB data, the head is segmented well (82% intersection over union, 88% accuracy), and performance is comparable with those achieved on large, public, non-neonatal datasets. On the other hand, performance on the NIR data was inferior. By employing data augmentation to generate additional virtual NIR data for training, results could be improved and the head could be segmented with 62% intersection over union and 65% accuracy. The method is in theory capable of performing segmentation in real time and thus it may provide a useful tool for future PPGI applications.

Distributed cutaneous tissue blood volume oscillations contain information on autonomic nervous system (ANS) regulation of cardiorespiratory activity as well as dominating thermoregulation. ANS associated with low-frequency oscillations can be quantified in terms of frequencies, amplitudes, and phase shifts. The relative order between these faculties may be disturbed by conditions colloquially termed ‘stress’. Photoplethysmography imaging, an optical non-invasive diagnostic technique provides information on cutaneous tissue perfusion in the temporal and spatial domains. Using the cold pressure test (CPT) in thirteen healthy volunteers as a well-studied experimental intervention, we present a method for evaluating phase shifts in low- and intermediate frequency bands in forehead cutaneous perfusion mapping. Phase shift changes were analysed in low- and intermediate frequency ranges from 0.05 Hz to 0.18 Hz. We observed that time waveforms increasingly desynchronised in various areas of the scanned area throughout measurements. An increase of IM band phase desynchronization observed throughout measurements was comparable in experimental and control group, suggesting a time effect possibly due to overshooting the optimal relaxation duration. CPT triggered an increase in the number of points phase-shifted to the reference that was specific to the low frequency range for phase-shift thresholds defined as π/4, 3π/8, and π/2 rad, respectively. Phase shifts in forehead blood oscillations may infer changes of vascular tone due to activity of various neural systems. We present an innovative method for the phase shift analysis of cutaneous tissue perfusion that appears promising to assess ANS change processes related to physical or psychological stress. More comprehensive studies are needed to further investigate the reliability and physiological significance of findings.

The early prediction of sepsis in intensive care units using clinical data is the objective of the Phys-ioNet/Computing in Cardiology Challenge 2019. In this paper, a machine learning approach is presented which uses an optimized Random Forest for prediction of a septic condition. After an initial data augmentation step, a customized learning process is performed for the trees to consider imbalance in the dataset. Finally, a feature reduction is implemented and the forest is trimmed to 50 trees for an optimal classification in terms of run time and accuracy. Using a 10-fold cross-validation on the complete training dataset, a mean utility score of 0.376 is achieved. In the final submission, a normalized observed utility score of 0.296 on the full test set is achieved. Our team name is The Septic Think Tank (final rank: 21).