Monitoring Cognitive Workload Using Vocal Tract and Voice Source Features

Abstract

IntroductionMonitoring cognitive workload in individuals that are performing safety-critical jobs has huge potential, for example in aviation [1]. The relationship between cognitive workload and performance has been well studied [2] and the connection between cognitive workload and physical health has also been highlighted [3]. Clearly, it is crucial to manage cognitive workload in the modern day work environment, both in terms of performance and health. Speech processing offers the ability to monitor cognitive workload in a non-intrusive way, compared to measures such as blood pressure, heart rate, electroencephalogram or electrocardiogram. Measuring voice is easy and recent work has shown very promising results in classifying cognitive workload levels using the speech signal [4][5][6]. Successful design and implementation of such a method would provide a powerful tool to developers of cognitive infocommunications systems [7].The main objective of this work is to provide an independent verification of whether there is a link between increased cognitive workload and changes in the speech signal and, if this link exists, to characterize what part of voice is mostly affected. To do this we conducted a cognitive workload experiment where the set tasks were primarily solved using speech from 98 participants. Each participant read a standard passage of text on a computer screen and solved Stroop tasks with three difficulty levels [8]. A separate classifier of the three difficulty levels was trained for each speaker based on two sets of voice parameters comprising of the vocal tract (VT) and voice source (VS) features respectively. The conclusion is that task load does affect the speech signal and that vocal tract parameters are a better indicator of task level than voice source parameters. The context of the work is presented in Section 2 and the speech processing approach is described in Section 3. The experimental methodology and detailed results are given in Section 4 and 5. The results are summarized in Section 6 and the discussion is continued in Section 7.

New Tools and Methods in Selection of Air Traffic Controllers Based on Multimodal Psychophysiological Measurements

“…A significant proportion of the identified features is already known to be related to sympathetic activity [16], [17], [19], [86], [87], [83], [84]. For example, F0 and loudness are expected to rise in the cases of heightened sympathetic activity [31], with changes in the whole spectral composition ''shifting'' to higher values.…”

“…From the set of speech features which were extracted, ranking of the candidates can be even intuitively interpreted, e.g., for a normally distributed sample of response times to Stroop tasks, participants with lower response times can be regarded as superior to those with higher response times. In the context of speech features responses to cognitive overload as a type of stress [16], [19], [86], [87], participants with lower LMD are exhibiting either lower sympathetic responses, or higher parasympathetic responses, or better balance of autonomic nervous system. Such individuals would be better suited for inherently highly stressful jobs, like pilots, ATCs etc.…”

“…Cognitive overload can be caused by different factors, such as variations of alertness, mental fatigue, mental effort, complexity of the task, attentional variations and drowsiness. In the context of ATC selection process cognitive load and its effects on speech features have already been researched [15]- [18], [19]. In our cognitive load experiment, we apply different versions of Stroop test, which have also been used in the context of selection and training of ATCs, e.g.…”

“…All the Stroop test variations were based on verbal task answers which additionally enables us to perform speech-based analyses of individual mental states like cognitive load/overload. The Stroop task was used in such a context in a number of previous papers, both in ATC samples and military applications [16], [17], [19], [83], [84], [86], [87].…”

Measuring Cognitive Load Using In-Game Metrics of a Serious Simulation Game

“…Behavioral measures rely on the analysis of differences in interaction behavior during task processing, such as speech and voice patterns ( Berthold and Jameson, 1999 ; Ruiz et al, 2010 ; Magnusdottir et al, 2017 ) or differences in the usage of input modalities such as keyboard or mouse ( Ikehara and Crosby, 2005 ; Lim et al, 2014 ). These measures are usually unobtrusive and do not distract participants from the task at hand.…”