A Multimodal System for Assessing Alertness Levels Due to Cognitive Loading

Abstract

This paper proposes a scheme for assessing the alertness levels of an individual using simultaneous acquisition of multimodal physiological signals and fusing the information into a single metric for quantification of alertness. The system takes electroencephalogram, high-speed image sequence, and speech data as inputs. Certain parameters are computed from each of these measures as indicators of alertness and a metric is proposed using a fusion of the parameters for indicating alertness level of an individual at an instant. The scheme has been validated experimentally using standard neuropsychological tests, such as the Visual Response Test (VRT), Auditory Response Test (ART), a Letter Counting (LC) task, and the Stroop Test. The tests are used both as cognitive tasks to induce mental fatigue as well as tools to gauge the present degree of alertness of the subject. Correlation between the measures has been studied and the experimental variables have been statistically analyzed using measures such as multivariate linear regression and analysis of variance. Correspondence of trends obtained from biomarkers and neuropsychological measures validate the usability of the proposed metric.

Mental fatigue measurement using eye metrics: A systematic literature review

“…Mental fatigue definitions are generally associated with reductions in performance (Herlambang et al., 2019; Hopstaken et al., 2015a, 2016, 2015; Li, Li et al., 2019; Sengupta et al., 2017), as also proven in 10 of the 34 studies. However, in practice, the feeling of fatigue is often caused by an attempt to maintain performance on the task (Burke et al., 2018; Di Stasi et al., 2015; Marandi et al., 2018), which would result in no or minimal variation in the overall performance.…”

“…Two metrics from the category of saccade—peak velocity and mean velocity—were inversely proportional to mental fatigue according to each of the eight studies that used them. Seven of the eight studies generated voluntary saccades using fatiguing tasks such as free visual exploration task (Cazzoli et al., 2014), driving (Di Stasi et al., 2012), aircraft flying (Di Stasi et al., 2016; LeDuc et al., 2005), cognitive tasks (Marandi et al., 2018; Renata et al., 2018; Sengupta et al., 2017). The remaining study involved pro‐saccades, a type of saccade generated by following a visual cue, in a gap‐overlap saccade task (Paladini et al., 2016).…”

“…Nine of the studies conducted multi-modal measurement of fatigue using EEG (Gergelyfi et al, 2015;Guo et al, 2018;Hopstaken et al, ,2015Hopstaken et al, , , 2015aHopstaken et al, , , 2016Sengupta et al, 2017;Wang et al, 2019), ECG (Abbasi et al, 2018;Horiuchi et al, 2018), and functional near-infrared spectroscopy (fNIRS; Borragán et al, 2018) and were able to confirm the corresponding measurements of mental fatigue using eyebased metrics.…”

“…The use of the fourth method of physiological signals and gaze behavior for mental fatigue measurement has increased in frequency, with the increasingly mobile and accurate technologies of electroencephalography (EEGGergelyfi et al., 2015; Guo et al., 2018; Hopstaken et al., ,2015, 2015a, 2016; Sengupta et al., 2017; Wang et al., 2019), electrocardiography (ECG; Abbasi et al., 2018; Horiuchi et al., 2018), EOG and eye‐tracking. Continuous fatigue measurement using these technologies provides data that is unhampered by subjective bias and does not require taking a break from an ongoing task.…”

Estimation and quantification of vigilance using ERPs and eye blink rate with a fuzzy model-based approach

“…In [34], Czisch et al used EEG and fMRI in liaison to study the effect of prolonged sleep deprivation on vigilance level. In [35], a system that combines information from multimodal data is presented to determine the state of alertness of an individual. Helton et al [36] examined the impact of task interruptions on vigilance.…”