Efficient Spatiotemporal Attention Network for Remote Heart Rate Variability Analysis

Abstract

Studies have shown that ordinary color cameras can detect the subtle color changes of the skin caused by the heartbeat cycle. Therefore, cameras can be used to remotely monitor the pulse in a non-contact manner. The technology for non-contact physiological measurement in this way is called remote photoplethysmography (rPPG). Heart rate variability (HRV) analysis, as a very important physiological feature, requires us to be able to accurately recover the peak time locations of the rPPG signal. This paper proposes an efficient spatiotemporal attention network (ESA-rPPGNet) to recover high-quality rPPG signal for heart rate variability analysis. First, 3D depth-wise separable convolution and a structure based on mobilenet v3 are used to greatly reduce the time complexity of the network. Next, a lightweight attention block called 3D shuffle attention (3D-SA), which integrates spatial attention and channel attention, is designed to enable the network to effectively capture inter-channel dependencies and pixel-level dependencies. Moreover, ConvGRU is introduced to further improve the network’s ability to learn long-term spatiotemporal feature information. Compared with existing methods, the experimental results show that the method proposed in this paper has better performance and robustness on the remote HRV analysis.

The Real-Time Image Sequences-Based Stress Assessment Vision System for Mental Health

“…The results showed that, at this stage, none of the participants demonstrated any signs of stress. For the random forest model trained on the SWELL dataset, an impressive detection rate was observed, with all 5 participants showing signs of stress As shown in Table 11, our proposed method performed better than traditional CHROM [18], FaceRPPG [19] methods, and deep learning-based PulseGAN [18], Phys-Net, [19], rPPGGAN [37], ESA-rPPGNet [19] methods. In the custom dataset, our proposed system showed an accuracy of 95% for HR and 82% for overall HRV features Table 12.…”

“…The results of these experiments are presented in Figures 7 and 8 and Table 11. As shown in Table 11, our proposed method performed better than traditional CHROM [18], FaceRPPG [19] methods, and deep learning-based PulseGAN [18], PhysNet, [19], rPPGGAN [37], ESA-rPPGNet [19] methods. In the custom dataset, our proposed system showed an accuracy of 95% for HR and 82% for overall HRV features Table 12.…”

“…Song et al [18] introduced the PulseGAN method, which incorporates CHROM and conditional generative adversarial networks (GAN), to estimate HRV from the face. Kuang et al [19] proposed ESA-rPPGNet, employing 3D depth-wise separable convolution to enhance network performance for HRV analysis. Some researchers used thermal images to extract features to detect stress from the face.…”

Evaluation of transformation invariant loss function with distance equilibrium in prediction of imaging photoplethysmography characteristics

“…A novel multitask temporal shift convolutional attention network (MTTS_CAN) was used to measure HR and Respiration rate (Resp), which leveraged Temporal Shift Models (Lin et al 2018) (TSMs) instead of 3D-CNN to perform efficient temporal modeling and uses the self-attention module (Liu et al 2020). An efficient spatiotemporal attention network (ESA-rPPGNet) was proposed to recover high-quality BVP signal for HRV, which adopted 3D depth-wise separable convolution and a structure based on mobile net v3 to greatly reduce the time complexity of the network (Kuang et al 2022). However, both Liu et al (2020) and Kuang et al (2022) only consider extracting spatiotemporal features from video and do not pay attention to recover the BVP signal shapes.…”

“…An efficient spatiotemporal attention network (ESA-rPPGNet) was proposed to recover high-quality BVP signal for HRV, which adopted 3D depth-wise separable convolution and a structure based on mobile net v3 to greatly reduce the time complexity of the network (Kuang et al 2022). However, both Liu et al (2020) and Kuang et al (2022) only consider extracting spatiotemporal features from video and do not pay attention to recover the BVP signal shapes. TS_CAN model is no difference from MTTS_CAN except that MTTS_CAN can predict BVP signals and Resp signal at the same time but TS_CAN can only predict BVP signal.…”