When do we interact multimodally?

Abstract

Mobile usage patterns often entail high and fluctuating levels of difficulty as well as dual tasking. One major theme explored in this research is whether a flexible multimodal interface supports users in managing cognitive load. Findings from this study reveal that multimodal interface users spontaneously respond to dynamic changes in their own cognitive load by shifting to multimodal communication as load increases with task difficulty and communicative complexity. Given a flexible multimodal interface, users' ratio of multimodal (versus unimodal) interaction increased substantially from 18.6% when referring to established dialogue context to 77.1% when required to establish a new context, a +315% relative increase. Likewise, the ratio of users' multimodal interaction increased significantly as the tasks became more difficult, from 59.2% during low difficulty tasks, to 65.5% at moderate difficulty, 68.2% at high and 75.0% at very high difficulty, an overall relative increase of +27%. Analysis of users' task-critical errors and response latencies across task difficulty levels increased systematically and significantly as well, corroborating the manipulation of cognitive processing load. The adaptations seen in this study reflect users' efforts to self-manage limitations on working memory when task complexity increases. This is accomplished by distributing communicative information across multiple modalities, which is compatible with a cognitive load theory of multimodal interaction. The long-term goal of this research is the development of an empirical foundation for proactively guiding flexible and adaptive multimodal system design.

The Paradigm Shift to Multimodality in Contemporary Computer Interfaces

“…The physical activity of manual or pen-based gesturing is believed 23 to play a particularly important role in organizing and facilitating people's spatial information processing, which has been shown to reduce cognitive load on tasks involving geometry, maps, and similar areas (Alibali et al, 2000;Oviatt, 1997) As task complexity increases, users self manage their working memory limits by distributing information across multiple modalities. As illustrated in Figure 3.3 (right), this represented an overall relative increase of +27% (Oviatt et al, 2004a). In one study, users' ratio of multimodal interaction increased as tasks became more difficult, from 59.2% during low-difficulty tasks, to 65.5% during moderate difficulty, 68.2% during high difficulty, and 75.0% on very high difficulty ones.…”

“…During more difficult tasks, this impact of gesturing on reducing cognitive load increased (Goldin-Meadow et al, 2001). This effectively enhances their performance during both perception and production tasks (Calvert et al, 2004;Mousavi et al, 1995;Oviatt, 1997;Oviatt et al, 2004a;Tang et al, 2005). This effectively enhances their performance during both perception and production tasks (Calvert et al, 2004;Mousavi et al, 1995;Oviatt, 1997;Oviatt et al, 2004a;Tang et al, 2005).…”

“…One important impact of this evolutionary transition to multimodal communication was to enable reduced cognitive load associated with communicating per se, which freed up human's mental resources for performing their primary tasks (Oviatt, 2013a;Oviatt et al, 2004a). As early hominids expanded their range of habitats and social contacts, improved multimodal communication and intelligibility helped them to adapt and thrive in a wide range of contexts.…”

“…There are large individual differences in users' multimodal interaction patterns, including their preference to interact unimodally vs. multimodally, and which modes they more strongly prefer (e.g., speaking vs. writing; Oviatt et al, 2004a). In research on speech and pen multimodal input to a computer, two distinct types of user integration patterns have been identified-simultaneous integrators, and sequential ones (see Figure 4.2).…”

Rate of speech and emotional-cognitive regulation in the psychotherapeutic process: a pilot study

“…More specifically, previous research has shown that higher levels of cognitive load result in a significantly lower ROS (Oviatt, Coulston, & Lunsford, 2004;Jameson, Kiefer, Muller, Großmann-Hutter, Wittig, & Rummer, 2010).…”