RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation

Abstract

Heart rate (HR) is an important physiological signal that reflects the physical and emotional status of a person. Traditional HR measurements usually rely on contact monitors, which may cause inconvenience and discomfort. Recently, some methods have been proposed for remote HR estimation from face videos; however, most of them focus on well-controlled scenarios, their generalization ability into less-constrained scenarios (e.g., with head movement, and bad illumination) are not known. At the same time, lacking large-scale HR databases has limited the use of deep models for remote HR estimation. In this paper, we propose an end-to-end RhythmNet for remote HR estimation from the face. In RyhthmNet, we use a spatial-temporal representation encoding the HR signals from multiple ROI volumes as its input. Then the spatial-temporal representations are fed into a convolutional network for HR estimation. We also take into account the relationship of adjacent HR measurements from a video sequence via Gated Recurrent Unit (GRU) and achieves efficient HR measurement. In addition, we build a large-scale multi-modal HR database (named as VIPL-HR 1 ), which contains 2,378 visible light videos (VIS) and 752 near-infrared (NIR) videos of 107 subjects. Our VIPL-HR database contains various variations such as head movements, illumination variations, and acquisition device changes, replicating a less-constrained scenario for HR estimation. The proposed approach outperforms the stateof-the-art methods on both the public-domain and our VIPL-HR databases.

DeepRhythm

“…A straightforward way of employing heart rate (HR) signals for DeepFake detection is to use existing HR representations that are designed for the remote HR estimation. For example, we can use the spatial-temporal representation (STR) proposed by Niu et al [48] for representing HR signals and feed them to a classifier for DeepFake detection. However, it is hard to achieve high fake detection accuracy with the STR directly since the differences between real and fake videos are not highlighted, i.e., STR's discriminative power for DeepFake detection is limited.…”

“…As shown in Table 3, our MMSTR significantly improves the DR-st's accuracy, e.g., 0.328 improvement on ALL. The ST map from [48] has little discriminative power for DeepFake detection since DR-st achieves about 0.5 accuracy on every testing dataset, which means it randomly guesses a video being real/fake. After using our MMSTR, DR-mmst achieves 0.217 averaged accuracy increment over DFD, DF, F2F, FS, and ALL datasets.…”

Non-Contact Heart Rate Detection When Face Information Is Missing during Online Learning

“…Therefore, a new publicly available dataset, directly related to rPPG-suitable practical applications, is vital. The currently available datasets include MAHNOB-HCI [33], DEAP [34], MMSE-HR [35], PURE [36], OBF [37], and VIPL-HR [38] et al, and their specifications are listed in Table 3. At present, rPPG is tentatively applied in Intensive Care Units (ICU), because the subjects are still and a frontal face video can be continuously collected.…”

Combining Photoplethysmography and Ballistocardiography to Address Voluntary Head Movements in Heart Rate Monitoring

“…To handle the noises from non-rigid motions, they divided the signal into smaller parts of the same length, computed the standard deviation for each subpart, and removed subparts having standard deviations too different from others. Finally, more recent studies are learning-based; they attempted to learn the mapping from facial videos to heart rate-related signals from a huge dataset using deep neural networks [34], [35].…”