Individual differences in multimodal integration patterns

Abstract

Techniques for information fusion are at the heart of multimodal system design. To develop new user-adaptive approaches for multimodal fusion, the present research investigated the stability and underlying cause of major individual differences that have been documented between users in their multimodal integration pattern. Longitudinal data were collected from 25 adults as they interacted with a map system over six weeks. Analyses of 1,100 multimodal constructions revealed that everyone had a dominant integration pattern, either simultaneous or sequential, which was 95-96% consistent and remained stable over time. In addition, coherent behavioral and linguistic differences were identified between these two groups. Whereas performance speed was comparable, sequential integrators made only half as many errors and excelled during new or complex tasks. Sequential integrators also had more precise articulation (e.g., fewer disfluencies), although their speech rate was no slower. Finally, sequential integrators more often adopted terse and direct command-style language, with a smaller and less varied vocabulary, which appeared focused on achieving error-free communication. These distinct interaction patterns are interpreted as deriving from fundamental differences in reflective-impulsive cognitive style. Implications of these findings are discussed for the design of adaptive multimodal systems with substantially improved performance characteristics.

Multimodal controls for soldier/swarm interaction

“…This use of commands indicates that individual touch-first and speech-first response patterns which were relatively consistent and stable over time. This agrees with the results found by Oviatt, Lunsford and Coulston [8], who also found consistent, stable individual differences in submitting commands in multimodal displays. However, when describing these individual differences, Oviatt, Lunsford and Coulston [8], and Oviatt, Coulston and Lunsford [7] noted that individual differences in timing of responses (sequential or simultaneous responses) were a dominant trend in individual differences, but did not describe any trend involving preferred modality of order of touch or speech output.…”

“…The time between the onset of a first control action (e.g., a speech command) relative to the onset of a second sequential, dependent control action (e.g., a consequent touch command) can be defined operationally as temporal binding, and is critical for the system to correctly interpret the operator's intent, as well as to support a smoother fusion of commands to the system and reduce system error. Although Oviatt, Lunsford and Coulston [8] suggested that identifying time between control actions is important, neither they nor any other researcher used sequential speech and touch controls. In addition, few researchers have explored user issues that might affect temporal binding.…”

“…In considering motor actions with touch screen displays, neither Oviatt [9], Oviatt, Lunsford and Coulston [8], nor any other researcher explored human performance using touch commands to input map locations with different levels of difficulty, such as static (nonmoving) versus dynamic (moving) touch screen targets (also referred to as map objects). In Oviatt's research, all map objects were static and none were differentiated by size.…”

“…User preference in the order of which modality (speech or touch) is used in commands (henceforth known as preferred modality) is important because multimodal systems usually have fixed, predefined temporal thresholds between commands [7,8]. These researchers noted that a system with a fixed temporal threshold that is too short might not integrate speech and touch signals correctly, causing the system to miss and/or eliminate the second part of a multimodal command, thus increasing system error.…”

Mixed reality participants in smart meeting rooms and smart home environments

“…When the system the user interacts with allows multimodality then more information about the user can be extracted in real-time. For example, the system may learn about interaction pattern preferences [1] or detect the user's emotional state and adapt its interaction behavior, its interface and its feedback according to them. The body and what the user is doing with his or her body is becoming important for the system and this is even more the case when the user is allowed to move around and interact from different positions and with various objects, maybe other users and parts of a computer-supported or monitored environment.…”