Attempt to extract features and classify subjective poor physical conditions in facial images using deep metric learning

Abstract

With the spread of COVID-19, the need for remote detection of physical conditions is increasing, for example, there are several situations wherein the body temperature has to be measured remotely to detect febrile individuals. Aiming to remotely detect physical conditions, the study attempted to investigate anomaly detection based on facial color and skin temperature, which are indicators related to hemodynamics. Triplet loss was used to extract features related to subjective health feelings from facial images to evaluate whether there is a relationship between subjective health feelings and facial images. A classification of subjective health feelings related to poor physical conditions based on these features was also attempted. To obtain the data, an experiment was conducted for approximately 1 year to measure facial visual and thermal images, and subjective feelings related to physical conditions. Anomaly levels were defined based on subjective health feelings. Anomaly detection models were constructed by classifying anomaly and normal data based on subjective health feelings. Facial visible and thermal images were applied to the trained model to quantitatively evaluate the accuracy of the classification of anomaly conditions related to subjective health. At higher levels of anomaly, a combination of facial visible and thermal images resulted in the classification of subjective health feelings with moderate accuracy. Further, the results suggest that the eyes and sides of the nose may indicate subjective health feelings.