Modeling multimodal integration patterns and performance in seniors

Abstract

Multimodal interfaces are designed with a focus on flexibility, although very few currently are capable of adapting to major sources of user, task, or environmental variation. The development of adaptive multimodal processing techniques will require empirical guidance from quantitative modeling on key aspects of individual differences, especially as users engage in different types of tasks in different usage contexts. In the present study, data were collected from fifteen 66-to 86-year-old healthy seniors as they interacted with a map-based flood management system using multimodal speech and pen input. A comprehensive analysis of multimodal integration patterns revealed that seniors were classifiable as either simultaneous or sequential integrators, like children and adults. Seniors also demonstrated early predictability and a high degree of consistency in their dominant integration pattern. However, greater individual differences in multimodal integration generally were evident in this population. Perhaps surprisingly, during sequential constructions seniors' intermodal lags were no longer in average and maximum duration than those of younger adults, although both of these groups had longer maximum lags than children. However, an analysis of seniors' performance did reveal lengthy latencies before initiating a task, and high rates of self talk and task-critical errors while completing spatial tasks. All of these behaviors were magnified as the task difficulty level increased. Results of this research have implications for the design of adaptive processing strategies appropriate for seniors' applications, especially for the development of temporal thresholds used during multimodal fusion. The long-term goal of this research is the design of highperformance multimodal systems that adapt to a full spectrum of diverse users, supporting tailored and robust future systems.

Multiple notification modalities and older users

“…Many researchers have called for systems that offer multiple modalities and are able to switch between them to suit the user, environment or message being delivered [2,6,7,9,12]. Research has shown that adaptable technology such as this can deliver notifications in a manner that is more acceptable Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

“…However, that work did not examine the differences between target and distractor notifications with older persons. As research suggests that older users have different expectations and requirements from technology than younger users [7,12], there is a need to further investigate multimodal reminder design for this group. This paper presents an experiment to investigate the disruptiveness and performance of different notification modalities when delivering target and distractor notifications to older users.…”

A Survey of Multimodal Systems and Techniques for Motor Learning

“… Intermodal lag in the sequential presentation of information [90] may cause significant delay issues for complex tasks, especially under numerous, rapid repetitions of a motor task.  Using different granularities of feedback for different categories can have undesired effects on the relative amount of motor learning in each domain.…”

Latent Semantic Analysis for Multimodal User Input With Speech and Gestures

“…without temporal overlap, denoted as "SEQ"). Further analysis shows that most of the subjects have adopted predominantly (with over 70% consistency) either simultaneous (87%, 20/23) or sequential (8.7%, 2/23) temporal patterns between speech and pen gestures [39], [40].…”