Multimodal behavior and interaction as indicators of cognitive load

Abstract

High cognitive load arises from complex time and safety-critical tasks, for example, mapping out flight paths, monitoring traffic, or even managing nuclear reactors, causing stress, errors, and lowered performance. Over the last five years, our research has focused on using the multimodal interaction paradigm to detect fluctuations in cognitive load in user behavior during system interaction. Cognitive load variations have been found to impact interactive behavior: by monitoring variations in specific modal input features executed in tasks of varying complexity, we gain an understanding of the communicative changes that occur when cognitive load is high. So far, we have identified specific changes in: speech, namely acoustic, prosodic, and linguistic changes; interactive gesture; and digital pen input, both interactive and freeform. As ground-truth measurements, galvanic skin response, subjective, and performance ratings have been used to verify task complexity. The data suggest that it is feasible to use features extracted from behavioral changes in multiple modal inputs as indices of cognitive load. The speech-based indicators of load, based on data collected from user studies in a variety of domains, have shown considerable promise. Scenarios include single-user and team-based tasks; think-aloud and interactive speech; and single-word, reading, and conversational speech, among others. Pen-based cognitive load indices have also been tested with some success, specifically with pen-gesture, handwriting, and freeform pen input, including diagraming. After examining some of the properties of these measurements, we present a multimodal fusion model, which is illustrated with quantitative examples from a case study. The feasibility of employing user input and behavior patterns as indices of cognitive load is supported by experimental evidence. Moreover, symptomatic cues of cognitive load derived from user behavior such as acoustic speech signals, transcribed text, digital pen trajectories of handwriting, and shapes pen, can be supported by well-established theoretical frameworks, including O'Donnell and Eggemeier's workload measurement [1986] Sweller's Cognitive Load Theory [Chandler and Sweller 1991], and Baddeley's model of modal working memory [1992] as well as McKinstry et al.'s [2008] and Rosenbaum's [2005] action dynamics work. The benefit of using this approach to determine the user's cognitive load in real time is that the data can be collected implicitly that is, during day-to-day use of intelligent interactive systems, thus overcomes problems of intrusiveness and increases applicability in real-world environments, while adapting information selection and presentation in a dynamic computer interface with reference to load.

A Survey of Multimodal Systems and Techniques for Motor Learning

“…This is especially true due to the time-sensitive or pressure-inducing nature of motor exercise tasks, as the user is attempting to maintain a certain pace, complete a number of repetitions within a time limit, or reach a certain progress goal [96].  Since information is being presented in parallel on three different aspects of a user's motion, selective attention toward one modality or domain may distract from learning in other domains.…”

A New Automated Method for Evaluating Mental Workload Using Handwriting Features

“…Although the empirical findings have demonstrated promise for using handwriting features to detect mental workload, there is only a limited understanding of how to use changes in handwriting behavior to indicate mental workload. For example, there is still no general agreement on what detailed handwriting features are predictive of mental workload [6]- [8], [14]. The studies by Yu et al [8] and Lin et al [9] reported that pressure-related features were good indicators of mental workload, while Luria and Rosenblum [7] did not find significant differences in pressure-related features under three mental workload conditions.…”

“…handwriting process and speech) can often be collected implicitly without extra equipment, thereby increasing their applicability in real-world environments. Recently, the feasibility of employing changes in handwriting and verbal behavior to indicate mental workload has been identified by a few empirical findings [6], [7]. In the study, we are interested in handwriting behavior because it is pervasive in current educational environments.…”

Cognitive Load Detection on Commercial EEG Devices: An Optimized Signal Processing Chain

“…Mental workload imposed on a person is an important component for human behavior modeling as it gives a direct representation of mental state of the person [1]. Cognitive load (CL) is the total amount of mental activity imposed on our working memory while doing any cognitive process.…”