Natural, Metaphoric, and Linguistic Auditory Direction Signals Have Distinct Influences on Visual Motion Processing

Abstract

To interact with our dynamic environment, the brain merges motion information from auditory and visual senses. However, not only "natural" auditory MOTION, but also "metaphoric" de/ascending PITCH and SPEECH (e.g., "left/right"), influence the visual motion percept. Here, we systematically investigate whether these three classes of direction signals influence visual motion perception through shared or distinct neural mechanisms. In a visual-selective attention paradigm, subjects discriminated the direction of visual motion at several levels of reliability, with an irrelevant auditory stimulus being congruent, absent, or incongruent. Although the natural, metaphoric, and linguistic auditory signals were equally long and adjusted to induce a comparable directional bias on the motion percept, they influenced visual motion processing at different levels of the cortical hierarchy. A significant audiovisual interaction was revealed for MOTION in left human motion complex (hMTϩ/V5ϩ) and for SPEECH in right intraparietal sulcus. In fact, the audiovisual interaction gradually decreased in left hMTϩ/V5ϩ for MOTION Ͼ PITCH Ͼ SPEECH and in right intraparietal sulcus for SPEECH Ͼ PITCH Ͼ MOTION. In conclusion, natural motion signals are integrated in audiovisual motion areas, whereas the influence of culturally learnt signals emerges primarily in higher-level convergence regions.

Visual motion aftereffect from understanding motion language

“…Another possibility is that understanding motion language recruits higher-level convergence areas that process visual motion, resulting in a bias to see dot motion in the same direction. Previous work has shown such a congruence effect in that hearing the words "right" and "left" biased participants to see an apparent motion stimulus as moving in the same direction (20). fMRI data revealed that this audiovisual interaction was driven more by activity in the anterior intraparietal sulcus than in hMT+.…”

Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes

“…Neuroimaging studies and transcranial magnetic stimulation (TMS) might help in future research to determine whether all of these correspondences are indeed on a par, neurologically speaking (see Bien et al, 2012;Sadaghiani et al, 2009;Spence & Parise, 2012).…”

“…There is also evidence that statistical crossmodal correspondences can modulate the neural response relatively early during information processing (that is, 220 ms after stimulus onset, see Bien, ten Oever, Goebel, & Sack, 2012;Spence & Parise, 2012;see Spence & Deroy, 2013, for a discussion). Such results should, of course, not be taken to imply that statistical correspondences do not also (or sometimes only) activate other loci later in information processing, as well (Sadaghiani, Maier, & Noppeney, 2009).…”

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

“…It is possible that crossmodal correspondences may differ in terms of their neural origin and, accordingly, manifest at different levels of the cognitive system (see also Sadaghiani, Maier, & Noppeney, 2009). Spence (2011) suggested that crossmodal correspondences can be classified into at three distinct types: structural, which most likely reflect direct correspondences in the neural processing of sensory information; statistical, which reflect crossmodal associations between sensory features or dimensions that exist in nature (e.g., the fact that small objects make higher-pitched sounds) and are, most likely, simply learned; and semantic, which apply when two dimensions overlap in the meaning (or associations) of the stimuli.…”