We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating handcrafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.

Abstract:The ability to monitor the respiratory rate, one of the vital signs, is extremely important for the medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake everyday activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Alternatively, contactless digital image sensor based remote-photoplethysmography (PPG) can be used. However, remote PPG requires an ambient source of light, and does not work properly in dark places or under varying lighting conditions. Recent advances in thermographic systems have shrunk their size, weight and cost, to the point where it is possible to create smart-phone based respiration rate monitoring devices that are not affected by lighting conditions. However, mobile thermal imaging is challenged in scenes with high thermal dynamic ranges (e.g. due to the different environmental temperature distributions indoors and outdoors). This challenge is further amplified by general problems such as motion artifacts and low spatial resolution, leading to unreliable breathing signals. In this paper, we propose a novel and robust approach for respiration tracking which compensates for the negative effects of variations in the ambient temperature and motion artifacts and can accurately extract breathing rates in highly dynamic thermal scenes. The approach is based on tracking the nostril of the user and using local temperature variations to infer inhalation and exhalation cycles. It has three main contributions. The first is a novel Optimal Quantization technique which adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking. The second is the Thermal Gradient Flow method that computes thermal gradient magnitude maps to enhance the accuracy of the nostril region tracking. Finally, we introduce the Thermal Voxel method to increase the reliability of the captured respiration signals compared to the traditional averaging method. We demonstrate the extreme robustness of our system to track the nostril-region and measure the respiratory rate by evaluating it during controlled respiration exercises in high thermal dynamic scenes (e.g. strong correlation (r = 0.9987) with the ground truth from the respiration-belt sensor). We also demonstrate how our algorithm outperformed standard algorithms in settings with different amounts of environmental thermal changes and human motion. We open the tracked ROI sequences of the datasets collected for these studies (i.e. under both controlled and unconstrained real-world settings) to the community to foster work in this area.

Background A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera–based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monitoring results, a person must remain still for several minutes while a measurement is being taken. This is cumbersome and makes its use in real-life situations impractical. Objective We proposed a system that combines PPG and thermography with the aim of improving cardiovascular signal quality and detecting stress responses quickly. Methods Using a smartphone camera with a low-cost thermal camera added on, we built a novel system that continuously and reliably measures 2 different types of cardiovascular events: (1) blood volume pulse and (2) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 participants, involved in stress-inducing mental workload tasks, measured their physiological responses to stressors over a short time period (20 seconds) immediately after each task. Participants reported their perceived stress levels on a 10-cm visual analog scale. For the instant stress inference task, we built novel low-level feature sets representing cardiovascular variability. We then used the automatic feature learning capability of artificial neural networks to improve the mapping between the extracted features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. Results First, we found that the measured PPG signals presented high quality cardiac cyclic information (mean pSQI: 0.755; SD 0.068). We also found that the measured thermal changes of the nose tip presented high-quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (mean pSQI: from 0.714 to 0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the cardiovascular signals (ie, heart rate variability and thermal directionality) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing precrafted features-based methods. In addition, the 17-fold leave-one-subject-out cross-validation results showed that combining both modalities produced higher accuracy than using PPG or thermal imaging only (PPG+Thermal: 78.33%; PPG: 68.53%; Thermal: 58.82%). The multimodal results are comparable to the state-of-the-art stress recognition methods that require long-term measurements. Finally, we explored effects of different data labeling strategies on the sensitivity of our inference methods. Our results showed the need for separation of and normalization between individual data. Conclusions The results demonstrate the feasibility of using smartphone-based imaging for instant stress detection. Given that this approach does not need long-term measurements requiring attention and reduced mobility, we believe it is more suitable for mobile mental health care solutions in the wild.

We report the experimental demonstration of an abnormal, opposite anti-crossing effect in a photon-magnon-coupled system that consists of an Yttrium Iron Garnet film and an inverted pattern of split-ring resonator structure (noted as ISRR) in a planar geometry. It is found that the normal shape of anti-crossing dispersion typically observed in photon-magnon coupling is changed to its opposite anti-crossing shape just by changing the position/orientation of the ISRR's split gap with respect to the microstrip line axis along which ac microwave currents are applied. Characteristic features of the opposite anti-crossing dispersion and its linewidth evolution are analyzed with the help of analytical derivations based on electromagnetic interactions. The observed opposite anti-crossing dispersion is ascribed to the compensation of both intrinsic damping and coupling-induced damping in the magnon modes. This compensation is achievable by controlling the relative strength and phase of oscillating magnetic fields generated from the ISRR's split gap and the microstrip feeding line. The position/orientation of an ISRR's split gap provides a robust means of controlling the dispersion shape of anti-crossing and its damping in a photon-magnon coupling, thereby offering more opportunity for advanced designs of microwave devices. a) Correspondence and requests for materials should be addressed to S.-K. K sangkoog@snu.ac.kr

No abstract