Thermal Imaging as a Way to Classify Cognitive Workload

Abstract

As epitomized in DARPA's 'Augmented Cognition' program, next generation avionics suites are envisioned as sensing, inferring, responding to and ultimately enhancing the cognitive state and capabilities of the pilot. Inferring such complex behavioural states from imagery of the face is a challenging task and multimodal approaches have been favoured for robustness. We have developed and evaluated the feasibility of a system for estimation of cognitive workload levels based on analysis of facial skin temperature. The system is based on thermal infrared imaging of the face, head pose estimation, measurement of the temperature variation across regions of the face and an artificial neural network classifier. The technique was evaluated in a controlled laboratory experiment using subjective measures of workload across tasks as a standard. The system was capable of accurately classifying mental workload into high, medium and low workload levels 81% of the time. The suitability of facial thermography for integration into a multimodal augmented cognition sensor suite is discussed.

Cognitive Heat

“…Thermal cameras are both unobtrusive and able to capture information from multiple users at a distance and at the same time. Previous research has shown that thermal patterns reveal different aspects of our internal states, including affect [24,60], stress [48], and deception [49]. Further, advances in miniaturization and mass production have continuously brought down the prices of these devices.…”

Thermal imaging based affective state recognition

“…The finding indicates there is a correlation between facial thermal features and affective states. Stemberger et aI., [7] reported on the development of Augmented Cognition program for next generation avionics suites with capability of measuring or classifying cognitive workload of the pilot. They concluded that the change in subjective and objective workload generated significant facial thermography signatures that could be acquired by thermal camera and further analyzed by the system.…”

Workload Alerts—Using Physiological Measures of Mental Workload to Provide Feedback During Tasks

“…Wilson and Russell [2003] presented different ways in which the physiological signals are known to change with the state of the operator: heart rate increases, for example, as the cognitive demands on the operator increase, and the rate of eye blinking decreases as the visual demands increases. Mental Workload changes, in particular, have been obtained by recording: cardiac activity [10,22,62], electrodermal response (EDR) and galvanic skin response (GSR) [14,50], eye function (pupil diameter) [9,32], imaging the brain [8,52], and even facial skin temperature [53]. As Mental Workload is a concept, each of these techniques essentially detect the changes in the autonomic nervous systems, which are then used to infer workload levels.…”

“…We used an fNIRS device, which has been shown to be suitable for HCI user study evaluations, but more commercially available devices like the NeuroSky 4 EEG device might be more suitable for day to day feedback. Similarly, even less invasive measures of mental workload could be taken from Heart Rate Variability [24] through smart watches, remotely detected by pupil dilation [33] or facial skin temperature [53] with cameras. Many of these other measures also better detect emotional responses, and perhaps concepts like stress and anxiety, and so might better serve future work on recording both Mental Workload and emotional response.…”