The Implications of Crossmodal Links in Attention for the Design of Multimodal Interfaces: A Driving Simulation Study

Abstract

The design of multimodal interfaces rarely takes into consideration recent data suggesting the existence of considerable crossmodal spatial and temporal links in attention. This can be partly explained by the fact that crossmodal links have been studied almost exclusively in spartan laboratory settings with simple cues and tasks. As a result, it is not clear whether they scale to more complex settings. To examine this question, participants in this experiment drove a simulated military vehicle and were periodically presented with lateralized visual indications marking locations of roadside mines and safe areas of travel. Valid and invalid auditory and tactile cues preceded these indications at varying stimulus-onset asynchronies. The findings confirm that the location and timing of crossmodal cue combinations affect response time and accuracy in complex domains as well. In particular, presentation of crossmodal cues at SOAs below 500ms and tactile cuing resulted in lower accuracy and longer response times.

Crossmodal spatial attention

“…Our results also argue against simple accounts of the modality‐specific nature of attentional resources 73–76 . However, these findings are not only of theoretical interest, they are also of profound applied relevance, given the growing interest in using nonvisual (and multisensory) cues to alert interface operators’ whose visual attention might otherwise be overloaded to signs of impending danger 77–80 …”

Comparison of Driver Brake Reaction Times to Multimodal Rear-end Collision Warnings

“…More research is necessary to determine the most effective placement, presentation, number, and intensity of a tactile display for rear-end collision warnings. Regardless of driver preference, the present findings, along with previous research (e.g., Lee, Hoffman, & Hayes, 2004;Ferris, Penfold, Hameed, & Sarter, 2006), support tactile warnings over visual and audio warnings for rear-end collision avoidance-some drivers may not have preferred the tactile warnings, but they worked the best to improve brake RT! Interestingly, there is a near-even split between drivers that prefer visual warnings and those that prefer tactile warnings.…”

Driving automation interface design : supporting drivers' changing role

“…acoustic, visual and haptic within concept E) caused an interrelated conflict. Ferris, Penfold, Hameed, and Sarter (2006) revealed that invalid cross-modal cues (e.g. a driver perceives an auditory cue to come from a different location than a visual cue although both are meant to indicate the same target) lead to longer response times and reduce response accuracy.…”

“…The acoustic warning did not provide spatial cues either. According to Ferris et al (2006) weak cross-modal links could already negatively influence performance. Therefore, weak crossmodal links of concept E could be an explanation for reduced reaction times.…”

“…Possible performance deficits due to unexpectedness of stimulus Ferris et al (2006) discovered that response times increase due to cues in an unexpected modality. Therefore, the unexpectedness of our tactile cue within a dominantly visual task could be an explanation for the low scores on intervention-support with concept E. An extreme manifestation when a stimulus appears unexpectedly is when it causes a startle response.…”