Data-driven machine learning models for decoding speech categorization from evoked brain responses

Abstract

Objective. Categorical perception (CP) of audio is critical to understand how the human brain perceives speech sounds despite widespread variability in acoustic properties. Here, we investigated the spatiotemporal characteristics of auditory neural activity that reflects CP for speech (i.e. differentiates phonetic prototypes from ambiguous speech sounds). Approach. We recorded 64-channel electroencephalograms as listeners rapidly classified vowel sounds along an acoustic-phonetic continuum. We used support vector machine classifiers and stability selection to determine when and where in the brain CP was best decoded across space and time via source-level analysis of the event-related potentials. Main results. We found that early (120 ms) whole-brain data decoded speech categories (i.e. prototypical vs. ambiguous tokens) with 95.16% accuracy (area under the curve 95.14%; F1-score 95.00%). Separate analyses on left hemisphere (LH) and right hemisphere (RH) responses showed that LH decoding was more accurate and earlier than RH (89.03% vs. 86.45% accuracy; 140 ms vs. 200 ms). Stability (feature) selection identified 13 regions of interest (ROIs) out of 68 brain regions [including auditory cortex, supramarginal gyrus, and inferior frontal gyrus (IFG)] that showed categorical representation during stimulus encoding (0–260 ms). In contrast, 15 ROIs (including fronto-parietal regions, IFG, motor cortex) were necessary to describe later decision stages (later 300–800 ms) of categorization but these areas were highly associated with the strength of listeners’ categorical hearing (i.e. slope of behavioral identification functions). Significance. Our data-driven multivariate models demonstrate that abstract categories emerge surprisingly early (∼120 ms) in the time course of speech processing and are dominated by engagement of a relatively compact fronto-temporal-parietal brain network.

Continuous dynamics in behavior reveal interactions between perceptual warping in categorization and speech-in-noise perception

“…First, we found mouse tracks diverged almost immediately (<100 ms) after stimulus presentation (Figure 4C), much earlier than listeners' collective RTs (∼800 ms). This is well within the timeframe (<250 ms) with which speech categories begin to emerge in auditory-sensory brain activity (Bidelman et al, 2013;Mahmud et al, 2021). Contextual effects due to stimulus history have also been observed in both animal (Lopez Espejo et al, 2019) and human (Carter et al, 2022) superior temporal gyrus.…”

Micro-state-based neural decoding of speech categorization using Bayesian non-parametrics

“…Intracranial sources underlying CP micro-states: These brain areas converge with both hypothesis and data-driven work which have shown a similar engagement of these regions in categorical decisions related to speech perception. For example, in our recent study (Mahmud, Yeasin, & Bidelman, 2021) applying machine learning techniques (e.g., neural classifiers, feature mining, stability selection) to full brain, source-reconstructed EEGs, we showed that 13 (out of 68) brain regions of the Desikan-Killiany (DK) (Desikan et al, 2006) ms). Indeed, microstates 1 and 9 identified via Bayesian non-parametric analysis of response RTs, isolate these patterned activities to nearly identical brain areas (STG, IFG; see Fig.…”

“…We replicate and extend these prior findings by demonstrating a similar neural network describing listeners' speed of categorical decision (i.e., RTs). It is worth noting that our microstate-based analysis here used an entirely different decoding approached applied to CP decision speeds (RTs) rather than listeners' binary labels of speech sounds as in previous studies (Mahmud et al, 2021). Yet, the consistency of our findings across divergent studies, methods, and behavioral assays is striking.…”

Self-supervised contrastive learning for EEG-based cross-subject motor imagery recognition

“…Frequently employed feature extraction algorithms include common spatial pattern, filter bank common spatial pattern (FBCSP), Fourier transform, power spectrum analysis, Wavelet transform, Autoregressive Model, and so forth [13][14][15][16][17]. Combining the features extracted by these algorithms with traditional classifiers such as linear discriminant analysis, support vector machines (SVM), and nearest neighbor classifiers has made essential contributions to decoding EEG signals [18,19]. However, achieving high decoding accuracy with such manual features is difficult since artificial experience or prior knowledge is added when extracting features.…”