Cross-modal selective attention: On the difficulty of ignoring sounds at the locus of visual attention

Abstract

In three experiments, we investigated whether the ease with which distracting sounds can be ignored depends on their distance from fixation and from attended visual events. In the first experiment, participants shadowed an auditory stream of words presented behind their heads, while simultaneously fixating visual lip-read information consistent with the relevant auditory stream, or meaningless "chewing" lip movements. An irrelevant auditory stream of words, which participants had to ignore, was presented either from the same side as the fixated visual stream or from the opposite side. Selective shadowing was less accurate in the former condition, implying that distracting sounds are harder to ignore when fixated. Furthermore, the impairment when fixating toward distractor sounds was greater when speaking lips were fixated than when chewing lips were fixated, suggesting that people find it particularly difficult to ignore sounds at locations that are actively attended for visual lipreading rather than merely passively fixated. Experiments 2 and 3 tested whether these results are specific to cross-modal links in speech perception by replacing the visual lip movements with a rapidly changing stream of meaningless visual shapes. The auditory task was again shadowing, but the active visual task was now monitoring for a specific visual shape at one location. A decrement in shadowing was again observed when participants passively fixated toward the irrelevant auditory stream. This decrement was larger when participants performed a difficult active visual task there versus fixating, but not for a less demanding visual task versus fixation. The implications for cross-modal links in spatial attention are discussed.The majority of selective-attention research has focused on selection within just one sensory modality. In many everyday situations, however, we need to coordinate our selective attention across different sensory modalities. For example, when listening to a conversation at a noisy gathering (the archetypal "cocktail party" situation), one must not only attend selectively to the sound of the speaker's voice but also integrate this auditory information with related visual information regarding the speaker's lip movements and gestures, while ignoring irrelevant events in both modalities.A number of audiovisual cross-modal links in spatial attention have now been demonstrated using variants of the spatial cuing methodology introduced by Posner (1978). For instance, Spence and Driver (1996) found that when people expected a visual target from one location, their per-

Sound source location modulates the irrelevant-sound effect

“…Auditory distractors are more difficult to ignore when they emanate from the same direction as that of the to-beattended visual stimuli (Spence et al, 2000). The experiments reported in the present article extend this finding to the irrelevant-sound effect.…”

“…Why is the effect so small in comparison with the huge effects observed by Spence et al (2000)? First, due to its retrospective, memory-based nature, serial recall performance may reflect effects of spatial attention to a much smaller degree (in comparison with other processes relevant for this performance measure) than would shadowing performance that is measured simultaneously with the distraction and presumably disrupts perceptual processes already.…”

“…The results are consistent with the assumption of cross-modal attentional links in models of attention, and they are problematic for explanations of the irrelevant-sound effect within working memory models that do not specify an explicit role of attention in the maintenance of information for immediate serial recall. Thus, investigating whether possible cross-modal links in spatial attention may be reflected in the irrelevant-sound effect has implications for models of working memoryin addition to the importance of such findings for models of cross-modal attention, as was pointed out by Spence et al (2000)-because attentional effects as such are informative about the validity of working memory models. In the present experiments, participants memorized sequences of digits while ignoring environmental sounds (Experiments 1 and 2) or spoken words (Experiment 3).…”

“…These data were interpreted as evidence in favor of models that posit cross-modal links in spatial attention in that apparently it was more difficult to ignore auditory distractors at visually attended locations than at some other point in space. Spence et al (2000) suggested that cross-modal links in spatial attention should also become apparent in other p phenomena, most notably the so-called irrelevant-sound effect-that is, an impairment in immediate serial recall t t of short lists of visually presented items when irrelevant auditory stimuli are presented either during encoding or during retention (Banbury, Tremblay, Macken, & Jones, 2001). Spence et al lamented a lack of studies investigating whether the size of the irrelevant-sound effect is d affected by the spatial proximity of the distractor sound source and the target display location "despite the fundamental importance of this issue for models of cross-modal attention" (p. 411).…”

“…In a series of experiments on cross-modal spatial attention (see Driver & Spence, 2004, for a review), Spence, Ranson, and Driver (2000) asked participants to shadow an auditory message from behind while attending a visual stream of events in one of the frontal hemifields. In addition to the to-be-shadowed auditory message from behind, an irrelevant stream of spoken words was displayed in front of the participants.…”