Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation

Abstract

To achieve the simultaneous and unobtrusive breathing rate (BR) and heart rate (HR) measurements during nighttime, we leverage a far-infrared imager and an infrared camera equipped with IR-Cut lens and an infrared lighting array to develop a dual-camera imaging system. A custom-built cascade face classifier, containing the conventional Adaboost model and fully convolutional network trained by 32K images, was used to detect the face region in registered infrared images. The region of interest (ROI) inclusive of mouth and nose regions was afterwards confirmed by the discriminative regression and coordinate conversions of three selected landmarks. Subsequently, a tracking algorithm based on spatio-temporal context learning was applied for following the ROI in thermal video, and the raw signal was synchronously extracted. Finally, a custom-made time-domain signal analysis approach was developed for the determinations of BR and HR. A dual-mode sleep video database, including the videos obtained under environment where illumination intensity ranged from 0 to 3 Lux, was constructed to evaluate the effectiveness of the proposed system and algorithms. In linear regression analysis, the determination coefficient (R2) of 0.831 had been observed for the measured BR and reference BR, and this value was 0.933 for HR measurement. In addition, the Bland-Altman plots of BR and HR demonstrated that almost all the data points located within their own 95% limits of agreement. Consequently, the overall performance of the proposed technique is acceptable for BR and HR estimations during nighttime.

Face Detection in Nighttime Images Using Visible-Light Camera Sensors with Two-Step Faster Region-Based Convolutional Neural Network

“…Because of these drawbacks, studies using NIR or SWIR cameras have been conducted [ 10 , 11 , 12 , 13 ]. NIR or SWIR cameras are robust to illumination changes and cost less than thermal cameras.…”

“…Lemoff et al [ 12 ] conducted tracking and recognition of people at a distance during day and night using their own tactical imager for night/day extended range surveillance (TINDERS) system. Hu et al [ 13 ] used a model that combines an adaboost and the fully convolutional network (FCN) [ 21 ] to detect the face of a person sleeping at night. In [ 22 ], they analyzed the performance of the Viola Jones face detector with NIR images.…”

“…Their experimental results show that the detection accuracy diminished according to the increase of standoff distance. The authors in [ 10 , 11 , 12 , 13 ] solved the problem of face detection at low intensity of illumination by using an NIR or SWIR camera. However, it is difficult to adaptively adjust the intensity and angle of the IR illuminator of such cameras.…”

“…To solve this problem, a near-infrared (NIR) light camera or a thermal camera has been used in most existing nighttime face detection studies [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ]. In contrast to the visible-light camera, NIR and thermal cameras use infrared (IR) wavelengths; therefore, they are robust to ambient lighting changes and low intensity of illumination.…”

Estimating Departure Time Using Thermal Camera and Heat Traces Tracking Technique

“…Far-infrared thermal imaging has been widely used in respiratory monitoring in telemedicine [10,11], obstacle detection [12,13] and pedestrian detection [14,15] in smart traffic, fault detection in smart industry [16,17], fire safety in smart cities [18,19], body temperature detection of pig in agriculture [20], etc. In criminal investigation, Pavlidis proposed that the facial image of a person could be obtained by thermal imaging technology, and then relevant analysis could be carried out to determine whether the person has fraudulent behavior [21].…”