Cross-Modal Links Among Vision, Audition, and Touch in Complex Environments

Abstract

Objectives: This study sought to determine whether performance effects of crossmodal spatial links that were observed in earlier laboratory studies scale to more complex environments and need to be considered in multimodal interface design. It also revisits the unresolved issue of cross-modal cuing asymmetries. Background: Previous laboratory studies employing simple cues, tasks, and/or targets have demonstrated that the efficiency of processing visual, auditory, and tactile stimuli is affected by the modality, lateralization, and timing of surrounding cues. Very few studies have investigated these cross-modal constraints in the context of more complex environments to determine whether they scale and how complexity affects the nature of crossmodal cuing asymmetries. Method: A microworld simulation of battlefield operations with a complex task set and meaningful visual, auditory, and tactile stimuli was used to investigate cuing effects for all cross-modal pairings. Results: Significant asymmetric performance effects of cross-modal spatial links were observed. Auditory cues shortened response latencies for collocated visual targets but visual cues did not do the same for collocated auditory targets. Responses to contralateral (rather than ipsilateral) targets were faster for tactually cued auditory targets and each visual-tactile cue-target combination, suggesting an inhibition-of-return effect. Conclusions: The spatial relationships between multimodal cues and targets significantly affect target response times in complex environments. The performance effects of cross-modal links and the observed cross-modal cuing asymmetries need to be examined in more detail and considered in future interface design. Application: The findings from this study have implications for the design of multimodal and adaptive interfaces and for supporting attention management in complex, data-rich domains.

Auditory, tactile, and multisensory cues facilitate search for dynamic visual stimuli

“…If participants' attention were to be automatically directed to the location of the tactile stimulation, as might have been the case for the auditory stimuli, this would mean that their attention would have been directed to their torsos and not to the computer moni-Whatever the correct explanation of the mechanism(s) underlying the effects reported here turns out to be, we believe that they have important implications for the design of nonvisual and multisensory warning signals in the automotive and air traffic management industries. Not only is the timing of the cue and target events vital in terms of producing significant performance improvements in tasks requiring the rapid detection and identification of potential threats (e.g., in air traffic management, pilot operations, vehicular navigation, collision avoidance, and military operations), but the spatial coincidence of the warning signal and target also seems to be necessary in order to produce warning signals that are maximally effective for these situations (Ferris & Sarter, 2008;Jones, Gray, Spence, & Tan, 2008;Sarter, 2000;). It appears that the benefits observed following the presentation of temporally synchronous and spatially informative cues result from the operation of somewhat different underlying mechanisms.…”

Tactile warning signals for in-vehicle systems

“…The presentation of directional tactile cues has been shown to lead to a shift of visual attention to the cued side, which may facilitate people's speeded responses when the direction of the tactile cues and the visual target coincide spatially (see Butter et al, 1989;Spence and Driver, 2004;Ho et al, 2005b;Ferris et al, 2006;Ngo and Spence, 2010;Gallace and Spence, 2014 though see Ferris and Sarter, 2008). In particular, researchers have typically studied the nature of the crossmodal links in spatial attention between vision and touch separately for the case of exogenous (see Spence et al, 1998;Kennett et al, 2001Kennett et al, , 2002 and endogenous (see Spence et al, 2000) spatial orienting (see Spence and Driver, 2004;Spence and Driver, 2004, for a review).…”

Effectiveness of Bimodal Versus Unimodal Alerts for Distracted Drivers

“…After giving written informed consent participants completed a brief demographic survey. Next, a C2 tactor was placed on the back of the participants left wrist and secured with an elastic wrist band, similar to methods used by (Ferris & Sarter, 2008). Participants then completed multiple practice sessions, including driving only, performing the n-back only at the 0-back and 3-back load levels and then performing both tasks at the same time.…”