On the similarities of representations in artificial and brain neural networks for speech recognition

Abstract

How the human brain supports speech comprehension is an important question in neuroscience. Studying the neurocomputational mechanisms underlying human language is not only critical to understand and develop treatments for many human conditions that impair language and communication but also to inform artificial systems that aim to automatically process and identify natural speech. In recent years, intelligent machines powered by deep learning have achieved near human level of performance in speech recognition. The fields of artificial intelligence and cognitive neuroscience have finally reached a similar phenotypical level despite of their huge differences in implementation, and so deep learning models can—in principle—serve as candidates for mechanistic models of the human auditory system. Utilizing high-performance automatic speech recognition systems, and advanced noninvasive human neuroimaging technology such as magnetoencephalography and multivariate pattern-information analysis, the current study aimed to relate machine-learned representations of speech to recorded human brain representations of the same speech. In one direction, we found a quasi-hierarchical functional organisation in human auditory cortex qualitatively matched with the hidden layers of deep neural networks trained in an automatic speech recognizer. In the reverse direction, we modified the hidden layer organization of the artificial neural network based on neural activation patterns in human brains. The result was a substantial improvement in word recognition accuracy and learned speech representations. We have demonstrated that artificial and brain neural networks can be mutually informative in the domain of speech recognition.Author summaryThe human capacity to recognize individual words from the sound of speech is a cornerstone of our ability to communicate with one another, yet the processes and representations underlying it remain largely unknown. Software systems for automatic speech-to-text provide a plausible model for how speech recognition can be performed. In this study, we used an automatic speech recogniser model to probe recordings from the brains of participants who listened to speech. We found that the parts of the dynamic, evolving representations inside the machine system were a good fit for representations found in the brain recordings, both showing similar hierarchical organisations. Then, we observed where the machine’s representations diverged from the brain’s, and made experimental adjustments to the automatic recognizer’s design so that its representations might better fit the brain’s. In so doing, we substantially improved the recognizer’s ability to accurately identify words.

On the similarities of representations in artificial and brain neural networks for speech recognition

“…Our findings imply that developing appropriate intermediate representations for articulatory features may be central to speech recognition in both human and machine solutions. In human neuroscience studies, this account is consistent with previous findings of articulatory feature representation in the human auditory cortex (Mesgarani et al, 2014;Correia et al, 2015;Wingfield et al, 2017), but awaits further investigation and exploitation in machine solutions for speech recognition. In particular, previous work by Hamilton et al (2021) has shown that-unlike our DNN architecture-the organization of early speech areas in the brain are not purely hierarchical, suggesting new potential avenues of model architectures including layerbypassing connections.…”

“…. Kriegeskorte et al, 2008b;Cadieu et al, 2014;Clarke et al, 2014;Khaligh-Razavi and Kriegeskorte, 2014;Güçlü and van Gerven, 2015;Kriegeskorte, 2015;Cichy et al, 2016;Kheradpisheh et al, 2016;Devereux et al, 2018), with less progress made in speech perception (though see our previous work: Su et al, 2014;Wingfield et al, 2017).…”

“…In order to track the cortical locations of brain-model correspondence, we estimated the location of cortical sources using the anatomically constrained MNE (Hämäläinen and Ilmoniemi, 1994) with identical parameters to those used in our previous work (Fonteneau et al, 2014;Su et al, 2014;Wingfield et al, 2017). MR structural images for each participant were obtained using a GRAPPA 3D MPRAGE sequence (TR = 2250 ms; TE = 2.99 ms; flip-angle = 9 deg; acceleration factor = 2) on a 3 T Trio (Siemens, Erlangen, Germany) with 1 mm isotropic voxels.…”

“…Heschl's gyrus (HG) and surrounding areas of the bilateral superior temporal cortices (STC) have also shown selective sensitivity to perceptual features of speech sounds earlier in the recognition process (Chan et al, 2014;Moerel et al, 2014;Saenz and Langers, 2014;Su et al, 2014;Thwaites et al, 2016). Building on our previous work investigating phonetic feature sensitivity in human auditory cortex (Wingfield et al, 2017), we focus our present analysis within language-related brain regions: STC and HG.…”

“…High-resolution magnetic resonance imaging (MRI) was acquired using a 3T MRI scanner for better source localization. As in our previous studies (Fonteneau et al, 2014;Su et al, 2014;Wingfield et al, 2017), the data (EMEG and MRI) has been combined to generate a source-space reconstruction of the electrophysiological activity which gave rise to the measurements at the electroencephalography (EEG) and MEG sensors. Using standard minimum-norm estimation (MNE) procedures guided by anatomical constraints from structural MRIs of the participants (Hämäläinen and Ilmoniemi, 1994;Gramfort et al, 2014), sources were localized to a cortical mesh at the gray-matter-white-matter boundary.…”