Early Cross-Modal Interactions in Auditory and Visual Cortex Underlie a Sound-Induced Visual Illusion

Mishra, Jyoti; Martı́nez, Antı́gona; Sejnowski, Terrence J.; Hillyard, Steven A.

doi:10.1523/jneurosci.4912-06.2007

Cited by 233 publications

(264 citation statements)

References 51 publications

Supporting

Mentioning

233

Contrasting

Order By: Relevance

“…The auditory-induced double-flash, a phenomenon wherein a single flash of light paired with two or more brief auditory events is perceived as multiple flashes of light (Shams et al, 2000;Mishra et al, 2007), has also been reported for the somatosensory modality (Violentyev et al, 2005), which correlates with activity in visual and somatosensory areas (Lange et al, 2011). In our paradigm (Fig.…”

Section: Stimuli and Proceduressupporting

confidence: 72%

Altered Cross-Modal Processing in the Primary Auditory Cortex of Congenitally Deaf Adults: A Visual-Somatosensory fMRI Study with a Double-Flash Illusion

Karns¹,

Dow²,

Neville³

2012

Journal of Neuroscience

167

152

View full text Add to dashboard Cite

The developing brain responds to the environment by using statistical correlations in input to guide functional and structural changesthat is, the brain displays neuroplasticity. Experience shapes brain development throughout life, but neuroplasticity is variable from one brain system to another. How does the early loss of a sensory modality affect this complex process? We examined cross-modal neuroplasticity in anatomically defined subregions of Heschl's gyrus, the site of human primary auditory cortex, in congenitally deaf humans by measuring the fMRI signal change in response to spatially coregistered visual, somatosensory, and bimodal stimuli. In the deaf Heschl's gyrus, signal change was greater for somatosensory and bimodal stimuli than that of hearing participants. Visual responses in Heschl's gyrus, larger in deaf than hearing, were smaller than those elicited by somatosensory stimulation. In contrast to Heschl's gyrus, in the superior-temporal cortex visual signal was comparable to somatosensory signal. In addition, deaf adults perceived bimodal stimuli differently; in contrast to hearing adults, they were susceptible to a double-flash visual illusion induced by two touches to the face. Somatosensory and bimodal signal change in rostrolateral Heschl's gyrus predicted the strength of the visual illusion in the deaf adults in line with the interpretation that the illusion is a functional consequence of the altered cross-modal organization observed in deaf auditory cortex. Our results demonstrate that congenital and profound deafness alters how vision and somatosensation are processed in primary auditory cortex.

show abstract

Section: Stimuli and Proceduressupporting

confidence: 72%

Altered Cross-Modal Processing in the Primary Auditory Cortex of Congenitally Deaf Adults: A Visual-Somatosensory fMRI Study with a Double-Flash Illusion

Karns¹,

Dow²,

Neville³

2012

Journal of Neuroscience

167

152

View full text Add to dashboard Cite

show abstract

“…One relevant point here is the integrative vs. segregative nature of the task with regard to multisensory processing: experimental paradigms using multisensory integration have classically reported an increase of sensory evoked responses, for instance when using the sound-induced flash illusion (Mishra et al, 2007;Watkins et al, 2006). In a TOJ task however, the segregation of auditory and visual information is a pre-requisite for successful ordering of auditory and visual events in time.…”

Section: Evoked Activity and Attention To Timementioning

confidence: 99%

Encoding of event timing in the phase of neural oscillations

2014

View full text Add to dashboard Cite

a b s t r a c t a r t i c l e i n f oTime perception is a critical component of conscious experience. To be in synchrony with the environment, the brain must deal not only with differences in the speed of light and sound but also with its computational and neural transmission delays. Here, we asked whether the brain could actively compensate for temporal delays by changing its processing time. Specifically, can changes in neural timing or in the phase of neural oscillation index perceived timing? For this, a lag-adaptation paradigm was used to manipulate participants' perceived audiovisual (AV) simultaneity of events while they were recorded with magnetoencephalography (MEG). Desynchronized AV stimuli were presented rhythmically to elicit a robust 1 Hz frequency-tagging of auditory and visual cortical responses. As participants' perception of AV simultaneity shifted, systematic changes in the phase of entrained neural oscillations were observed. This suggests that neural entrainment is not a passive response and that the entrained neural oscillation shifts in time. Crucially, our results indicate that shifts in neural timing in auditory cortices linearly map participants' perceived AV simultaneity. To our knowledge, these results provide the first mechanistic evidence for active neural compensation in the encoding of sensory event timing in support of the emergence of time awareness.

show abstract

“…If that is the case, we would expect familiar and unfamiliar objects to be processed differently in the context of the sound-induced flash illusion. The sound-induced flash illusion represents a good way to test the role of visual familiarity in early audio-visual interactions because the illusion originates early in the processing stream [21,27,30,35] and because audition, not vision, dominates in this task [28][29].…”

Section: Page 5 Of 21mentioning

confidence: 99%

“…Integration of the audio-visual stimuli of the flash illusion appears to occur at early stages of stimulus processing [21,27,30,35] and it is linked to activity in the visual cortex (V1) although it is unclear whether the activity in V1 is due to direct inputs from the primary auditory cortex or to back-projections from the superior temporal polysensory area [35]. As for the time course of familiar stimuli processing, evidence from face recognition studies shows that recognising a face as familiar involves a series of processes [3], including the processing of visual features of familiar individuals, then the unique semantic information associated to the person, and then his/her name (see also [16]).…”

Section: Page 5 Of 21mentioning

confidence: 99%

See 1 more Smart Citation

Familiarity of objects affects susceptibility to the sound-induced flash illusion

Setti

Chan

2011

Neuroscience Letters

View full text Add to dashboard Cite

A c c e p t e d M a n u s c r i p t  Visual familiarity reduces susceptibility to the sound induced flash-illusion  The effect of familiarity holds for faces and buildings  Early audio-visual multisensory interactions are modulated by high level processing AbstractAudition is accepted as more reliable (thus dominant) than vision when temporal discrimination is required by the task. However, it is not known whether the characteristics of the visual stimulus, for example its familiarity to the perceiver, affect auditory dominance. In this study we manipulated familiarity of the visual stimulus in a well-established multisensory phenomenon, i.e. the sound-induced flash illusion. This illusion occurs when, for example, one brief visual stimulus (e.g. a flash) is presented in close temporal proximity with two brief sounds; participants perceive two flashes instead of one. We found that when the visual faces or buildings stimuli were familiar, participants were less susceptible to the illusion than when they were unfamiliar. As the illusion has been ascribed to early cross-sensory interactions between vision and audition, the present work offers behavioural evidence that high level processing of objects" characteristics such as familiarity, affect early temporal multisensory integration. Possible mechanisms underlying the effect of familiarity are discussed.

show abstract

Early Cross-Modal Interactions in Auditory and Visual Cortex Underlie a Sound-Induced Visual Illusion

Cited by 233 publications

References 51 publications

Altered Cross-Modal Processing in the Primary Auditory Cortex of Congenitally Deaf Adults: A Visual-Somatosensory fMRI Study with a Double-Flash Illusion

Altered Cross-Modal Processing in the Primary Auditory Cortex of Congenitally Deaf Adults: A Visual-Somatosensory fMRI Study with a Double-Flash Illusion

Encoding of event timing in the phase of neural oscillations

Familiarity of objects affects susceptibility to the sound-induced flash illusion

Contact Info

Product

Resources

About