Advancing Non-Contact Vital Sign Measurement using Synthetic Avatars

McDuff, Daniel; Hernández, Javier; Wood, Erroll; Liu, Xin; Baltrušaitis, Tadas

doi:10.48550/arxiv.2010.12949

Cited by 6 publications

(10 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our proposed method cannot completely close the gap, although we believe that our approach is a step in the right direction; (3) We are aware that we have a non-uniform distribution of skin types in this dataset, and the same is true for many other PPG datasets [29]. Specifically, we only have eight participants with the skin type of VI and V. Recent efforts have been made to address these imbalances, but these data are not publicly available at this time [4,25]. We plan to expand our dataset with better coverage of skin types.…”

Section: Limitationsmentioning

confidence: 91%

MobilePhys: Personalized Mobile Camera-Based Contactless Physiological Sensing

Liu,

Wang,

Xie

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Camera-based contactless photoplethysmography refers to a set of popular techniques for contactless physiological measurement. The current state-of-the-art neural models are typically trained in a supervised manner using videos accompanied by gold standard physiological measurements. However, they often generalize poorly out-of-domain examples (i.e., videos that are unlike those in the training set). Personalizing models can help improve model generalizability, but many personalization techniques still require some gold standard data. To help alleviate this dependency, in this paper, we present a novel mobile sensing system called MobilePhys, the first mobile personalized remote physiological sensing system, that leverages both front and rear cameras on a smartphone to generate high-quality self-supervised labels for training personalized contactless camera-based PPG models. To evaluate the robustness of MobilePhys, we conducted a user study with 39 participants who completed a set of tasks under different mobile devices, lighting conditions/intensities, motion tasks, and skin types. Our results show that MobilePhys significantly outperforms the state-of-the-art on-device supervised training and few-shot adaptation methods. Through extensive user studies, we further examine how does MobilePhys perform in complex real-world settings. We envision that calibrated or personalized camera-based contactless PPG models generated from our proposed dual-camera mobile sensing system will open the door for numerous future applications such as smart mirrors, fitness and mobile health applications.

show abstract

Section: Limitationsmentioning

confidence: 91%

MobilePhys: Personalized Mobile Camera-Based Contactless Physiological Sensing

Liu,

Wang,

Xie

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Every participant was instructed to maintain stationary for the first two tasks, and then to perform head motions with increasing rotational velocity in the next four tasks (turning from left to right). Along with AFRL dataset, we also leverage a synthetic avatar video dataset introduced by McDuff et al (2020) where each synthetic video is parameterized and generated with a custom pulse signal, background, facial appearance, and motion. More specifically, the input pulse signal is used to augment skin color and the subsurface radius of skin pixels to mimic the effect of the blood volume pulse on the skin's appearance.…”

Section: Methodsmentioning

confidence: 99%

EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement

Liu

Hill

Jiang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Camera-based physiological measurement is a growing field with neural models providing state-the-art-performance. Prior research have explored various "endto-end" models; however these methods still require several preprocessing steps. These additional operations are often non-trivial to implement making replication and deployment difficult and can even have a higher computational budget than the "core" network itself. In this paper, we propose two novel and efficient neural models for camera-based physiological measurement called EfficientPhys that remove the need for face detection, segmentation, normalization, color space transformation or any other preprocessing steps. Using an input of raw video frames, our models achieve state-of-the-art accuracy on three public datasets. We show that this is the case whether using a transformer or convolutional backbone. We further evaluate the latency of the proposed networks and show that our most light weight network also achieves a 33% improvement in efficiency.

show abstract

“…Generative modeling definitely offers many opportunities in physiological measurement [80,135]. Computer graphics can provide a way to create high fidelity videos of the human body with augmented motions and skin subsurface scattering that simulate cardiac and respiratory processes [88].…”

Section: Synthetics and Data Augmentationmentioning

confidence: 99%

Camera Measurement of Physiological Vital Signs

McDuff¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The need for remote tools for healthcare monitoring has never been more apparent. Camera measurement of vital signs leverages imaging devices to compute physiological changes by analyzing images of the human body. Building on advances in optics, machine learning, computer vision and medicine these techniques have progressed significantly since the invention of digital cameras. This paper presents a comprehensive survey of camera measurement of physiological vital signs, describing they vital signs that can be measured and the computational techniques for doing so. I cover both clinical and non-clinical applications and the challenges that need to be overcome for these applications to advance from proofs-of-concept. Finally, I describe the current resources (datasets and code) available to the research community and provide a comprehensive webpage (https://cameravitals.github.io/) with links to these resource and a categorized list of all the papers referenced in this article. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing; • Applied computing → Bioinformatics; • Computing methodologies → Computer vision.

show abstract

Advancing Non-Contact Vital Sign Measurement using Synthetic Avatars

Cited by 6 publications

References 40 publications

MobilePhys: Personalized Mobile Camera-Based Contactless Physiological Sensing

MobilePhys: Personalized Mobile Camera-Based Contactless Physiological Sensing

EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement

Camera Measurement of Physiological Vital Signs

Contact Info

Product

Resources

About