Deep speech-to-text models capture the neural basis of spontaneous speech in everyday conversations

Abstract

Humans effortlessly use the continuous acoustics of speech to communicate rich linguistic meaning during everyday conversations. In this study, we leverage 100 hours (half a million words) of spontaneous open-ended conversations and concurrent high-quality neural activity recorded using electrocorticography (ECoG) to decipher the neural basis of real-world speech production and comprehension. Employing a deep multimodal speech-to-text model named Whisper, we develop encoding models capable of accurately predicting neural responses to both acoustic and semantic aspects of speech. Our encoding models achieved high accuracy in predicting neural responses in hundreds of thousands of words across many hours of left-out recordings. We uncover a distributed cortical hierarchy for speech and language processing, with sensory and motor regions encoding acoustic features of speech and higher-level language areas encoding syntactic and semantic information. Many electrodes including those in both perceptual and motor areas display mixed selectivity for both speech and linguistic features. Notably, our encoding model reveals a temporal progression from language-to-speech encoding before word onset during speech production and from speech-to-language encoding following word articulation during speech comprehension. This study offers a comprehensive account of the unfolding neural responses during fully natural, unbounded daily conversations. By leveraging a multimodal deep speech recognition model, we highlight the power of deep learning for unraveling the neural mechanisms of language processing in real-world contexts.

Aligning Brains into a Shared Space Improves their Alignment to Large Language Models

“…Many recent studies have begun to employ encoding models to predict neural responses during natural language processing using contextual embeddings derived from LLMs (Schrimpf et al, 2021;Caucheteux & King, 2022;Goldstein et al, 2022;Toneva et al, 2022;Cai et al, 2023;Goldstein, Wang, et al, 2023;Mischler et al, 2024;Zada et al, 2023). Our study demonstrates that aligning the neural activity in each brain into a shared, stimulus-driven feature space significantly enhances encoding performance.…”

“…The vast majority of prior work fitting electrode-wise linguistic encoding models does not evaluate whether models generalize across individual subjects (Goldstein et al 2022;Goldstein, Wang, et al 2023;Mischler et al 2024;cf. Zada et al 2023).…”

“…This simple approach of linearly "aligning" the LLM's internal feature space to human brain features has yielded remarkably good prediction performance in both functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG). The high spatiotemporal resolution of invasive ECoG recordings, in particular, promises to provide finer-grained insights into shared representations and processes between LLMs and the brain (Goldstein et al, 2022;Cai et al, 2023;Goldstein, Wang, et al, 2023;Mischler et al, 2024;Zada et al, 2023;Goldstein et al, 2024).…”

Neural populations in the language network differ in the size of their temporal receptive windows

“…Several previous studies have probed intracranial neural responses during language comprehension (e.g., Fedorenko et al, 2016; Nelson et al, 2017; Woolnough et al, 2023; Desbordes et al, 2023; Goldstein et al, 2022; 2023). For example, Fedorenko et al (2016) reported sensitivity in language-responsive electrodes to both word meanings and combinatorial processing, in line with fMRI findings (e.g., Fedorenko et al, 2010; Bedny et al, 2011).…”