PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference Transformer

Yu, Zitong; Shen, Yuming; Shi, Jingang; Zhao, Hengshuang; Cui, Yawen; Zhang, Jiehua; Torr, Philip H. S.; Zhao, Guoying

doi:10.1007/s11263-023-01758-1

Cited by 47 publications

(6 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the statistical significance of many results fell short of the expected level, highlighting the difficulty in acquiring high-quality real-world data and the challenge in ROI detection for both r-PPG and thermal imaging in more realistic settings. Future studies should compare and explore other techniques for improving prediction accuracy, including state-of-the-art machine learning models ( Lu, Han & Zhou, 2021 ; Yu et al, 2023 ). This would make the proposed method more practically useful.…”

Section: Discussionmentioning

confidence: 99%

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Liu,

Zhong

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

Multimodal emotion recognition techniques are increasingly essential for assessing mental states. Image-based methods, however, tend to focus predominantly on overt visual cues and often overlook subtler mental state changes. Psychophysiological research has demonstrated that heart rate (HR) and skin temperature are effective in detecting autonomic nervous system (ANS) activities, thereby revealing these subtle changes. However, traditional HR tools are generally more costly and less portable, while skin temperature analysis usually necessitates extensive manual processing. Advances in remote photoplethysmography (r-PPG) and automatic thermal region of interest (ROI) detection algorithms have been developed to address these issues, yet their accuracy in practical applications remains limited. This study aims to bridge this gap by integrating r-PPG with thermal imaging to enhance prediction performance. Ninety participants completed a 20-min questionnaire to induce cognitive stress, followed by watching a film aimed at eliciting moral elevation. The results demonstrate that the combination of r-PPG and thermal imaging effectively detects emotional shifts. Using r-PPG alone, the prediction accuracy was 77% for cognitive stress and 61% for moral elevation, as determined by a support vector machine (SVM). Thermal imaging alone achieved 79% accuracy for cognitive stress and 78% for moral elevation, utilizing a random forest (RF) algorithm. An early fusion strategy of these modalities significantly improved accuracies, achieving 87% for cognitive stress and 83% for moral elevation using RF. Further analysis, which utilized statistical metrics and explainable machine learning methods including SHapley Additive exPlanations (SHAP), highlighted key features and clarified the relationship between cardiac responses and facial temperature variations. Notably, it was observed that cardiovascular features derived from r-PPG models had a more pronounced influence in data fusion, despite thermal imaging’s higher predictive accuracy in unimodal analysis.

show abstract

Section: Discussionmentioning

confidence: 99%

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Liu,

Zhong

et al. 2024

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…In more recent studies (Yu et al 2022(Yu et al , 2023, researchers delved into the field of rPPG by claiming that recent deep learning approaches have primarily focused on extracting subtle rPPG cues using CNNs with limited spatio-temporal receptive fields (figure 9). Consequently, these methods tend to overlook the importance of long-range spatio-temporal perception and interaction in rPPG modeling.…”

Section: Deep Learning-based Methods For Hr Estimationmentioning

confidence: 99%

Conventional and deep learning methods in heart rate estimation from RGB face videos

Helwan,

Azar,

Ma’aitah

2024

Physiol. Meas.

View full text Add to dashboard Cite

Contactless Vital Signs Monitoring is a fast advancing scientific field aims to employ monitoring methods that do not necessitate the use of leads or physical attachments to the patient in order to overcome the shortcomings and limits of traditional monitoring systems. Several traditional methods have been applied to extract the heart rate (HR) signal from the face. Moreover, machine learning has recently contributed majorly to the development of such field in which deep networks and other deep learning methods were employed to extract the heart rate signal from RGB face videos. In this paper, we evaluate the state-of-the-art conventional and deep learning methods for heart rate estimate, focusing on the limits of deep learning methods and the availability of less-controlled face video datasets. We aim to present an extensive review that helps understanding the various approaches of remote Photoplethysmography (rPPG) extraction and HR estimation in addition to their drawbacks and benefits.

show abstract

“…Yu et al also introduced Physformer++ [71] utilizing a SlowFast temporal difference transformer with two pathways, and incorporates periodic-and cross-attention mechanisms. In contrast to PhysFormer, which only employs the slow pathway, Physformer++ uses both pathways to leverage temporal contextual and periodic rPPG clues from facial videos more effectively.…”

Section: Transformer Networkmentioning

confidence: 99%

Review on remote heart rate measurements using photoplethysmography

Lee,

Sivakumar,

Lim

2023

Multimed Tools Appl

View full text Add to dashboard Cite

Remote photoplethysmography (rPPG) gains recent great interest due to its potential in contactless heart rate measurement using consumer-level cameras. This paper presents a detailed review of rPPG measurement using computer vision and deep learning techniques for heart rate estimation. Several common gaps and difficulties of rPPG development are highlighted for the feasibility study in real-world applications. Numerous computer vision and deep learning methods are reviewed to mitigate crucial issues such as motion artifact and illumination variation. In comparison, deep learning approaches are proven more accurate than conventional computer vision methods due to their adaptive pattern learning and generalization characteristics. An increasing trend of applying deep learning techniques in rPPG can improve effective heart rate estimation and artifact removal. To consider more realistic disturbances into account, additional vital signs and large training datasets are crucial to improve the accuracy of heart rate estimations. By taking the benefit of contactless and accurate estimation, the application of rPPG can be greatly adopted in real-world activities, especially in precision sports.

show abstract

PhysFormer++: Facial Video-Based Physiological Measurement with SlowFast Temporal Difference Transformer

Cited by 47 publications

References 74 publications

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Conventional and deep learning methods in heart rate estimation from RGB face videos

Review on remote heart rate measurements using photoplethysmography

Contact Info

Product

Resources

About