Relating dynamic brain states to dynamic machine states: human and machine solutions to the speech recognition problem

Abstract

There is widespread interest in the relationship between the neurobiological systems supporting human cognition and emerging computational systems capable of emulating these capacities. Human speech comprehension, poorly understood as a neurobiological process, is an important case in point. Automatic Speech Recognition (ASR) systems with nearhuman levels of performance are now available, which provide a computationally explicit solution for the recognition of words in continuous speech. This research aims to bridge the gap between speech recognition processes in humans and machines, using novel multivariate techniques to compare incremental 'machine states', generated as the ASR analysis progresses over time, to the incremental 'brain states', measured using combined electroand magneto-encephalography (EMEG), generated as the same inputs are heard by human listeners. This direct comparison of dynamic human and machine internal states, as they respond to the same incrementally delivered sensory input, revealed a significant correspondence between neural response patterns in human superior temporal cortex and the structural properties of ASR-derived phonetic models. Spatially coherent patches in human temporal cortex responded selectively to individual phonetic features defined on the basis of machine-extracted regularities in the speech to lexicon mapping process. These results demonstrate the feasibility of relating human and ASR solutions to the problem of speech recognition, and suggest the potential for further studies relating complex neural computations in human speech comprehension to the rapidly evolving ASR systems that address the same problem domain. Author summaryThe ability to understand spoken language is a defining human capacity. But despite decades of research, there is still no well-specified account of how sound entering the ear is neurally interpreted as a sequence of meaningful words. At the same time, modern computer-based Automatic Speech Recognition (ASR) systems are capable of nearhuman levels of performance, especially where word-identification is concerned. In this research we aim to bridge the gap between human and machine solutions to speech recognition. We use a novel combination of neuroimaging and statistical methods to relate human and machine internal states that are dynamically generated as spoken words are heard by human listeners and analysed by ASR systems. We find that the stable regularities discovered by the ASR process, linking speech input to phonetic labels, can be significantly related to the regularities extracted in the human brain. Both systems may have in common a representation of these regularities in terms of articulatory phonetic features, consistent with an analysis process which recovers the articulatory gestures that generated the speech. These results suggest a possible partnership between human-and machinebased research which may deliver both a better understanding of how the human brain provides such a robust solution to speech understanding, an...

On the similarities of representations in artificial and brain neural networks for speech recognition

“…Our findings imply that developing appropriate intermediate representations for articulatory features may be central to speech recognition in both human and machine solutions. In human neuroscience studies, this account is consistent with previous findings of articulatory feature representation in the human auditory cortex [13,37,63], but awaits further investigation and exploitation in machine solutions for speech recognition. Recently, large deep artificial neural network models pre-trained on a massive amount of unlabelled waveform features (e.g.…”

“…This has primarily been in the domain of visual object perception (e.g. [7,11,12,17,22,[27][28][29]34]), with less progress made in speech perception (though see our previous work; [56,63]).…”

“…In order to track the cortical locations of brain-model correspondence, we estimated the location of cortical sources using the anatomically constrained MNE [23] with identical parameters to those used in [19,56,63]. MR structural images for each participant were obtained using a 16 GRAPPA 3D MPRAGE sequence (TR = 2250 ms; TE = 2.99 ms; flip-angle = 9 deg; acceleration factor = 2) on a 3 T Trio (Siemens, Erlangen, Germany) with 1 mm isotropic voxels.…”

“…Our hypothesis was that the representations in auditory cortex would be organised according to phones and features [63]. To investigate how the assignment of phonetic and featural labels to each segment of the stimuli could explain hidden-layer representations in DNN-BN 7 , we computed Davies-Bouldin clustering indices for representational spaces at each layer.…”

“…Heschl’s gyrus (HG) and surrounding areas of the bilateral superior temporal cortices (STC) have also shown selective sensitivity to perceptual features of speech sounds earlier in the recognition process [8, 40, 50, 56, 57]. Building on our previous work investigating phonetic feature sensitivity in human auditory cortex [63], we focus our present analysis within language-related brain regions: STC and HG.…”