Visual cortex responds to sound onset and offset during passive listening

Abstract

Sounds enhance our ability to detect, localize, and respond to co-occurring visual targets. Research suggests that sounds improve visual processing by resetting the phase of ongoing oscillations in visual cortex. However, it remains unclear what information is relayed from the auditory system to visual areas and if sounds modulate visual activity even in the absence of visual stimuli (e.g., during passive listening). Using intracranial electroencephalography (iEEG) in humans, we examined the sensitivity of visual cortex to three forms of auditory information during a passive listening task: auditory onset responses, auditory offset responses, and rhythmic entrainment to sounds. Because some auditory neurons respond to both sound onsets and offsets, visual timing and duration processing may benefit from each. Additionally, if auditory entrainment information is relayed to visual cortex, it could support the processing of complex stimulus dynamics that are aligned between auditory and visual stimuli. Results demonstrate that in visual cortex, amplitude-modulated sounds elicited transient onset and offset responses in multiple areas, but no entrainment to sound modulation frequencies. These findings suggest that activity in visual cortex (as measured with iEEG in response to auditory stimuli) may not be affected by temporally fine-grained auditory stimulus dynamics during passive listening (though it remains possible that this signal may be observable with simultaneous auditory-visual stimuli). Moreover, auditory responses were maximal in low-level visual cortex, potentially implicating a direct pathway for rapid interactions between auditory and visual cortices. This mechanism may facilitate perception by time-locking visual computations to environmental events marked by auditory discontinuities.

Early visual cortex tracks speech envelope in the absence of visual input

“…Overall, our results align with recent evidence reporting that the visual cortex can contribute to auditory information processing in sighted individuals (Brang et al, 2022; Martinelli et al, 2020; Seydell-Greenwald et al, 2021; Vetter et al, 2014). Here, we observed that the visual cortex is more engaged in processing when speech signal is intelligible and clear (i.e., presented in quiet).…”

“…The contribution of visual cortices in language processing is not limited to visual or audiovisual representations of spoken language. There is scattered evidence that the early visual cortex is also active during purely auditory stimulation (Brang et al, 2022; Petro, Paton, & Muckli, 2017; Vetter, Smith, & Muckli, 2014) and while listening to spoken language (e.g., Martinelli et al, 2020; Seydell-Greenwald, Wang, Newport, Bi, & Striem-Amit, 2021; Wolmetz, Poeppel, & Rapp, 2011). Importantly, such activations cannot be explained by semantic-based imagery alone but rather seem to reflect genuine responses to language input; in fact, the visual cortex also responds to abstract concepts with low imaginability rates (Seydell-Greenwald et al, 2021).…”

Effects of rhythmic auditory stimulation on vision: Oscillations in performance can be enhanced, but not induced.

“…However, without concurrent physiological recording, it is difficult to generalize from “absence of perceptual modulations” to “absence of physiological entrainment.” It is at least plausible that alpha-band oscillatory activity could synchronize weakly to auditory stimulation without inducing robust modulations in behavior. The suggestive but ultimately weak and inconsistent results of Experiments 1 and 4 could be consistent with such weak synchronization, but we note that Brang et al (2022) failed to observe neural oscillations in visual cortex in response to a similar rhythmic auditory stimulus.…”

“…There is always the possibility that our failure to observe auditory to visual entrainment is a matter of the choice of stimulus, task, or other design element unrelated to the process in question and that a more effective design could succeed in inducing or revealing visual perceptual oscillations where we have failed. However, we expect these results (the failure to induce stable oscillations in visual perception) to generalize to other types of auditory entraining stimuli and similar visual tasks, particularly given corroborating evidence of failures to induce either auditory perceptual oscillations or visual cortical oscillations in this range of frequencies using other types of entraining stimuli (Farahbod et al, 2020; and Brang et al, 2022; respectively).…”

“…In these contexts, as in the present study, it is unclear whether subjects allocate attention to the auditory stimulus. In a recent study 1168 CUNNINGHAM, WANG, AND BECK using intracranial EEG in humans, Brang et al (2022) suggest that passive auditory stimulation is not sufficient to induce robust visual entrainment, at least at frequencies beyond 5 Hz (frequencies below 5 Hz were not probed). In their design, individuals listened to amplitude modulated auditory stimuli with no concurrent visual task, so the experimental contexts are not identical, but the results are suggestive and consistent with our observations.…”