DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks

Abstract

Non-contact video-based physiological measurement has many applications in health care and human-computer interaction. Practical applications require measurements to be accurate even in the presence of large head rotations. We propose the first end-to-end system for videobased measurement of heart and breathing rate using a deep convolutional network. The system features a new motion representation based on a skin reflection model and a new attention mechanism using appearance information to guide motion estimation, both of which enable robust measurement under heterogeneous lighting and major motions. Our approach significantly outperforms all current state-of-the-art methods on both RGB and infrared video datasets. Furthermore, it allows spatial-temporal distributions of physiological signals to be visualized via the attention mechanism.

RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation

“…Hsu et al [9] generated time-frequency maps from the pre-processed green channel signals and used them as input of a VGG-16 model to estimate the HR. Chen et al [26] fed the original RGB face video into a convolutional network with attention mechanism and output the related HR signals. Although the existing data-driven approaches attempted to make use of statistical learning, as opposed to physical model based signal analysis, they failed to build an end-to-end HR estimator.…”

Architectural Tricks for Deep Learning in Remote Photoplethysmography

“…ECG is quoted to be more reliable [28], while contact PPG is quoted to be closer to rPPG signal retrieved [38], making training easier. Reproduction of ground truth signal is often the main area of ML-based training [3].…”

“…Some mask is often applied to select the so-called "region of interest" (ROI) -area of the frame image without background pixels and with most informative fluctuations [3,17,20]. In most cases, the video of the face is processed.…”

“…To overcome this problem, transfer learning approach is used with original data coming from other domains [5,6] or even generated out from training on mock signals [18]. Transfer learning capability is also used in [3] to measure quality of the proposed method.…”

“…Facial bounding box is detected by the OpenCV implementation of the Viola-Jones face detector [32] applied to each frame. Regarding the works on ROIs selection [11,12], six ROIs are used in this paper: r = 1..6 ( (1), nose bridge (2), areas under eyes (3,4), truncated facial box (5), full bounding box (6). ROIs coordinates are not ideally symmetrical about a horizontal axis: they were set manually to match the facial features on most video sequences in the dataset.…”