Self-supervised models of audio effectively explain human cortical responses to speech

Abstract

Self-supervised language models are very effective at predicting high-level cortical responses during language comprehension. However, the best current models of lower-level auditory processing in the human brain rely on either hand-constructed acoustic filters or representations from supervised audio neural networks. In this work, we capitalize on the progress of self-supervised speech representation learning (SSL) to create new stateof-the-art models of the human auditory system. Compared against acoustic baselines, phonemic features, and supervised models, representations from the middle layers of self-supervised models (APC, wav2vec, wav2vec 2.0, and HuBERT) consistently yield the best prediction performance for fMRI recordings within the auditory cortex (AC). Brain areas involved in low-level auditory processing exhibit a preference for earlier SSL model layers, whereas higher-level semantic areas prefer later layers. We show that these trends are due to the models' ability to encode information at multiple linguistic levels (acoustic, phonetic, and lexical) along their representation depth. Overall, these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

“…However, the overall variance explained was very low. Similarly, Vaidya et al 29 demonstrated that certain self-supervised speech models capture distinct stages of speech processing. Our results complement these findings in showing that they apply to a large set of models and to responses to natural sounds more generally.…”

“…It is nonetheless conceivable (and perhaps likely) that fully accurate models will require learning algorithms that more closely resemble the optimization processes of biology, in which nested loops of evolutionary selection and (largely unsupervised) learning over development combine to produce systems that can perform a wide range of tasks with breathtaking accuracy and efficiency. Some steps in this direction can be found in recent models that are optimized without labeled training data 27, 29, 86, 87 . Our model set contained one such contrastive self-supervised model (Wav2vec2), and although its brain predictions were worse than those of most of the supervised models, this direction clearly merits extensive exploration.…”

“…Comparisons of human and model behavioral characteristics have found that audio-trained neural networks often reproduce patterns of human behavior when optimized for naturalistic tasks and stimulus sets [15][16][17][18][19] . Several studies have also compared audio-trained neural networks to brain responses within the auditory system 20,15,[21][22][23][24][25][26][27] .…”

“…Comparisons of human and model behavioral characteristics have found that audio-trained neural networks often reproduce patterns of human behavior when optimized for naturalistic tasks and stimulus sets [17][18][19][20][21] . Several studies have also compared audio-trained neural networks to brain responses within the auditory system 22,17,[23][24][25][26][27][28][29] . The best-known of these prior studies is arguably that of , who found that DNNs jointly optimized for speech and music classification could predict functional magnetic resonance imaging (fMRI) responses to natural sounds in auditory cortex substantially better than a standard model based on spectrotemporal filters.…”

“…Several subsequent studies built on these findings by analyzing models trained on various speech-related tasks, and found they were able to predict cortical responses to speech better than chance, with some evidence that different model stages best predicted different brain regions 26,28,27,29 . But each of these studies analyzed only a small number of models, and each used a different brain data set, making it difficult to compare results across studies, and leaving the generality of brain-DNN similarities unclear.…”