Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain

Abstract

The extent to which brain functions are localized or distributed is a foundational question in neuroscience. In the human brain, common fMRI methods such as cluster correction, atlas parcellation, and anatomical searchlight are biased by design toward finding localized representations. Here we introduce the functional searchlight approach as an alternative to anatomical searchlight analysis, the most commonly used exploratory multivariate fMRI technique. Functional searchlight removes any anatomical bias by grouping voxels based only on functional similarity and ignoring anatomical proximity. We report evidence that visual and auditory features from deep neural networks and semantic features from a natural language processing model are more widely distributed across the brain than previously acknowledged. This approach provides a new way to evaluate and constrain computational models with brain activity and pushes our understanding of human brain function further along the spectrum from strict modularity toward distributed representation.

Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech

“…By focusing on speech representations, the present study provides four empirical contributions to the investigation of auditory representations in brains and deep learning models (Yamins & DiCarlo, 2016;Keshishian et al, 2020;Berezutskaya et al, 2020;Khosla et al, 2020;Kell et al, 2018;Kumar et al, 2020;Koumura et al, 2019).…”

“…speech-to-text using Dutch, English or Bengali). Previous studies have already shown that deep convolutional neural networks trained to classify words, musical genres (Kell et al, 2018;Kumar et al, 2020) or natural sounds (Koumura et al, 2019), generate brain-like representations, in the sense that one can find a linear correspondence between the activation of the neural networks and the activations of the brain (Figure 1). This similarity can be quantified with a "brain score" (Yamins et al, 2014), a correlation between the brain measurements and a linear projection of the model's activations, under the assumption that representations are defined as linearly exploitable information (Hung et al, 2005;Kamitani & Tong, 2005;Kriegeskorte & Kievit, 2013;King et al, 2018).…”

BrainIAK: The Brain Imaging Analysis Kit

“…We previously validated that this fitting procedure produces accurate simulations of real data [23]. We have used fmrisim to evaluate the power of different experimental design parameters [23], and also to evaluate the efficacy of new analysis methods [78,79].…”

Uncovering the Topology of Time-Varying fMRI Data using Cubical Persistence

“…SRMs 'learn' a mapping of multiple subjects into the same space, enabling the detection of group differences, or the study of relations between brain activity and movie annotations, for example [54]. Recently, SRM was used to map voxel activity into a functional space (as opposed to an anatomical one), to study the brain representation of, among others, visual and auditory information while receiving naturalistic audiovisual stimuli [29]. Nonetheless, while SRM is one of the most powerful techniques for extracting cognitively-relevant signals from fMRI data, there is still room for improvement.…”