On the similarities of representations in artificial and brain neural networks for speech recognition

Abstract

IntroductionIn recent years, machines powered by deep learning have achieved near-human levels of performance in speech recognition. The fields of artificial intelligence and cognitive neuroscience have finally reached a similar level of performance, despite their huge differences in implementation, and so deep learning models can—in principle—serve as candidates for mechanistic models of the human auditory system.MethodsUtilizing high-performance automatic speech recognition systems, and advanced non-invasive human neuroimaging technology such as magnetoencephalography and multivariate pattern-information analysis, the current study aimed to relate machine-learned representations of speech to recorded human brain representations of the same speech.ResultsIn one direction, we found a quasi-hierarchical functional organization in human auditory cortex qualitatively matched with the hidden layers of deep artificial neural networks trained as part of an automatic speech recognizer. In the reverse direction, we modified the hidden layer organization of the artificial neural network based on neural activation patterns in human brains. The result was a substantial improvement in word recognition accuracy and learned speech representations.DiscussionWe have demonstrated that artificial and brain neural networks can be mutually informative in the domain of speech recognition.

Continuous Hopfield dynamics of semantic vector-based activation patterns predict variations of priming effects over items

“…There thus appears to be a potential for elegant and constructive theoretical reductionism over multiple psychological tasks and phenomena. In line with this, artificial neural networks even appear to capture some aspects of how the brain processes information at the neural level (Hao et al, 2020;Jocham et al, 2011;Suri & Schultz, 1999;Wingfield et al, 2022).…”

Hearing what is being said: The distributed neural substrate for early speech interpretation

“…We see this, for example, in the Kocagoncu et al (2017) RSA analyses for the same dataset, which showed well circumscribed and differentiated patches of model fit for the different lexical and semantic models tested, while Lyu et al (2019) use MEG/ssRSA to reveal in unprecedented neurocognitive detail the neural dynamics of lexical semantic interpretation in sentential contexts. In earlier papers focusing on acoustic and phonological processes in medial and lateral STG, we use MEG/ssRSA techniques to pick out spatiotemporally well-defined patches of model fit corresponding to different subsets of auditory input analyses (Su et al, 2014;Wingfield et al, 2017Wingfield et al, , 2022. For the analyses reported here, there are several examples of spatially differentiated model fit -for example left and right angular gyrus and right middle temporal pole all show strong fit to the phonology model but no fit to the entropy or semantics models (Figure 3 and Table 1).…”