Camera fusion for real-time temperature monitoring of neonates using deep learning

Abstract

The continuous monitoring of vital signs is a crucial aspect of medical care in neonatal intensive care units. Since cable-based sensors pose a potential risk for the immature skin of preterm infants, unobtrusive monitoring techniques using camera systems are increasingly investigated. The combination of deep learning–based algorithms and camera modalities such as RGB and infrared thermography can improve the development of cable-free methods for the extraction of vital parameters. In this study, a real-time approach for local extraction of temperatures on the body surface of neonates using a multi-modal clinical dataset was implemented. Therefore, a trained deep learning–based keypoint detector was used for body landmark prediction in RGB. Image registration was conducted to transfer the RGB points to the corresponding thermographic recordings. These landmarks were used to extract the body surface temperature in various regions to determine the central-peripheral temperature difference. A validation of the keypoint detector showed a mean average precision of 0.82. The registration resulted in mean absolute errors of 16.4 px (8.2 mm) for x and 22.4 px (11.2 mm) for y. The evaluation of the temperature extraction revealed a mean absolute error of 0.55 $$^{\circ }$$ ∘ C. A final performance of 31 fps was observed on the NVIDIA Jetson Xavier NX module, which proves real-time capability on an embedded GPU system. As a result, the approach can perform real-time temperature extraction on a low-cost GPU module. Graphical abstract

A non-invasive method to determine core temperature for cats and dogs using surface temperatures based on machine learning

“…In another study, rectal temperatures of rabbits were predicted using measurements obtained with advanced infrared cameras [ 23 ]. There were also reports for human infants, among them, Lyra et al combined deep learning-based algorithms and camera modalities to real-time monitor the temperature of neonates [ 25 ]; Yaeger et al developed a natural language processing algorithm to identify febrile infants [ 26 ]; Asano et al applied a semantic segmentation method to thermal images, which makes it possible to monitor the temperature distribution over the whole body of infants [ 27 ]. Different to those approaches in predicting core temperatures, our method focused on companion animals using convenient operations and equipment.…”

Multi-modal body part segmentation of infants using deep learning

“…The dataset used in this work was recorded by Lyra et al [ 29 ] in the NICU of Saveetha Medical College & Hospital in Chennai, India. The study was approved by the institutional ethics committee of Saveetha University (SMC/IEC/2018/03/067).…”

Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset

“…Written informed consent was obtained from the parents of all patients. The recordings were performed with a high-end multicamera setup, for which a detailed description can be found in [ 26 ]. In total, 2850 images of 19 different stable newborns (150 frames per patient) were subsampled from this dataset.…”

“…Further, since also infrared images were recorded in the described study, more advanced fusion algorithms based on GANs [ 45 ] could be used for the generation of condition images. In order to evaluate the actual usage of the generated data for augmentation, the images will be used in the training steps of different DL approaches such as segmentation [ 46 ], and body pose estimation [ 26 ]. Here, the performance change of using the data in the training step and the prediction score of using the images as test data will be investigated.…”