Auditory features modelling reveals sound envelope representation in striate cortex

Abstract

Primary visual cortex is no longer considered exclusively visual in its function. Proofs that its activity plays a role in multisensory processes have accrued. Here we provide evidence that, in absence of retinal input, V1 maps sound envelope information. We modeled amplitude changes occurring at typical speech envelope timescales of four hierarchically-organized categories of natural (or synthetically derived) sounds. Using functional magnetic resonance, we assessed whether sound amplitude variations were represented in striate cortex and, as a control, in the temporal cortex. Sound amplitude mapping in V1 occurred regardless of the semantic content, was dissociated from the spectral properties of sounds and was not restricted to speech material. As in the temporal cortex, a spatially organized representation of amplitude modulation frequencies emerged in V1. Our results demonstrate that human striate cortex is a locus of representation of sound attributes.

A modality independent proto-organization of human multisensory areas

“…We took advantage of computational modeling to extract a collection of movie-related features. Specifically, two sets of low-level features were defined, one extracted from the auditory stream (spectral 26 and sound envelope 53 properties to account for frequency- and amplitude-based modulations) and one from the visual stream (set of static Gabor-like filters -GIST 54,55 and motion energy information based on their spatiotemporal integration 56 ). Moreover, a second set of high-level features was modeled based on a manual tagging of natural and artificial categories occurring both in the auditory and visual streams, as well as word embeddings from subtitles using the word2vec algorithm 57 .…”

Early visual cortex tracks speech envelope in the absence of visual input

“…Importantly, such activations cannot be explained by semantic-based imagery alone but rather seem to reflect genuine responses to language input; in fact, the visual cortex also responds to abstract concepts with low imaginability rates (Seydell-Greenwald et al, 2021). Overall, this evidence highlights a putative role of the visual cortex in mapping temporal modulations of incoming sounds, especially in the absence of competing retinal input (Martinelli et al, 2020;Vetter et al, 2014). However, the exact role of the visual cortex in the hierarchy of speech processing remains unclear.…”

“…Visual ROIs were selected for the left and right hemispheres and included primary (V1; Calcarine sulcus) and secondary (V2, Lingual gyrus) visual cortex, defined as the 'S_calcarine' and the 'G_oc-tem_med-Lingual' scouts in the atlas, correspondingly. These visual ROIs were selected based on recently reported evidence of their involvement in speech processing not only in blind but also sighted 13 individuals, albeit to a lower extent (Martinelli et al, 2020;Petro et al, 2017;Seydell-Greenwald et al, 2021;Van Ackeren, Barbero, Mattioni, Bottini, & Collignon, 2018;Vetter et al, 2020Vetter et al, , 2014. Upon the ROIs creation, their time-series were extracted and submitted to the analysis.…”

“…The copyright holder for this preprint this version posted June 28, 2022. ; https://doi.org/10.1101/2022.06.28.497713 doi: bioRxiv preprint temporal modulations of incoming sounds, especially in the absence of competing retinal input (Martinelli et al, 2020;Vetter et al, 2014). However, the exact role of the visual cortex in the hierarchy of speech processing remains unclear.…”

“…The contribution of visual cortices in language processing is not limited to visual or audiovisual representations of spoken language. There is scattered evidence that the early visual cortex is also active during purely auditory stimulation (Brang et al, 2022; Petro, Paton, & Muckli, 2017; Vetter, Smith, & Muckli, 2014) and while listening to spoken language (e.g., Martinelli et al, 2020; Seydell-Greenwald, Wang, Newport, Bi, & Striem-Amit, 2021; Wolmetz, Poeppel, & Rapp, 2011). Importantly, such activations cannot be explained by semantic-based imagery alone but rather seem to reflect genuine responses to language input; in fact, the visual cortex also responds to abstract concepts with low imaginability rates (Seydell-Greenwald et al, 2021).…”