Joint Encoding of Auditory Timing and Location in Visual Cortex

Abstract

Co-occurring sounds can facilitate perception of spatially and temporally correspondent visual events. Separate lines of research have identified two putatively distinct neural mechanisms underlying two types of crossmodal facilitations: Whereas crossmodal phase resetting is thought to underlie enhancements based on temporal correspondences, lateralized occipital evoked potentials (ERPs) are thought to reflect enhancements based on spatial correspondences. Here, we sought to clarify the relationship between these two effects to assess whether they reflect two distinct mechanisms or, rather, two facets of the same underlying process. To identify the neural generators of each effect, we examined crossmodal responses to lateralized sounds in visually responsive cortex of 22 patients using electrocorticographic recordings. Auditory-driven phase reset and ERP responses in visual cortex displayed similar topography, revealing significant activity in pericalcarine, inferior occipital–temporal, and posterior parietal cortex, with maximal activity in lateral occipitotemporal cortex (potentially V5/hMT+). Laterality effects showed similar but less widespread topography. To test whether lateralized and nonlateralized components of crossmodal ERPs emerged from common or distinct neural generators, we compared responses throughout visual cortex. Visual electrodes responded to both contralateral and ipsilateral sounds with a contralateral bias, suggesting that previously observed laterality effects do not emerge from a distinct neural generator but rather reflect laterality-biased responses in the same neural populations that produce phase-resetting responses. These results suggest that crossmodal phase reset and ERP responses previously found to reflect spatial and temporal facilitation in visual cortex may reflect the same underlying mechanism. We propose a new unified model to account for these and previous results.

Visual cortex responds to transient sound changes without encoding complex auditory dynamics

“…However, contrary to these alternatives, the functional properties of pulvinar neurons and their cortical connections are largely inconsistent with the iEEG responses observed in our experiments using lateralized sounds (Plass et al, 2019). For example, consistent with lateralized blindsight phenomena in humans, inactivation of the pulvinar in non-human primates with V1 lesions selectively impaired visuallyguided saccades in the contralesional hemifield, while completely sparing the ipsilateral hemifield (Kinoshita et al, 2019).…”

“…By contrast, in our iEEG experiments, sounds lateralized at approximately 45 degrees away from the midline produced bilateral responses throughout visual cortex, with only a modest contralateral bias (Plass et al, 2019). Thus, the topography of crossmodal responses to lateralized sounds is not consistent with the expected strictly-hemispheric response pattern predicted by a pulvinar source.…”

“…Direct connections between low-level auditory and visual cortices may also contribute to the facilitative effects of spatialized sounds on visual perception and spatial orienting by modulating visual activity in a hemispatially biased manner (e.g., Brang et al, 2015;Plass et al, 2019).…”

“…While early investigations focused on multisensory convergence in the superior colliculus and higher-order association cortices (Stein and Meredith, 1993;Ghazanfar and Schroeder, 2006), recent research has demonstrated the importance of crossmodal modulations of oscillatory activity in nominally "unisensory" cortical areas (Kayser and Logothetis, 2007;Thorne and Debener, 2014;van Atteveldt et al, 2014;Murray et al, 2016) such as sounds evoking early responses in primary visual cortex (Brang et al, 2015;Plass et al, 2019). By crossmodally entraining or resetting the phase of oscillations in visual cortex, simple transient sounds can enhance cortical sensitivity for temporally coupled visual input (Lakatos et al, 2009;Naue et al, 2011;Romei et al, 2012;Mercier et al, 2013Mercier et al, , 2015.…”

Selective attention to sound features mediates cross-modal activation of visual cortices

“…It remains unresolved, however, to what extent spatial representations are truly “visual”, even if localised within nominally visual cortices, or instead are multisensory both in their content and functional consequences on perception and behaviour (see Ten Oever et al, 2016 for discussion as well as Eimer et al, 2002; Gamble & Luck, 2011). A further possibility is that cross-modal lateralised ERP differences at the occipital scalp and in visual cortices are instead a result of a carryover effect of lateralised processing of contralateral vs. ipsilateral sound stimuli occurring first in auditory cortices, which is subsequently modulated by top-down attentional control mechanisms (Plass et al, 2019). Such an account is (at least partially) supported by evidence of spatial attention effects on early-latency lateralised ERP components presumed to originate within auditory cortices (Hillyard et al, 1973; Alho et al, 1999).…”