From speech to thought: the neuronal basis of cognitive units in non-experimental, real-life communication investigated using ECoG

Abstract

Exchange of thoughts by means of expressive speech is fundamental to human communication. However, the neuronal basis of real-life communication in general, and of verbal exchange of ideas in particular, has rarely been studied until now. Here, our aim was to establish an approach for exploring the neuronal processes related to cognitive “idea” units (IUs) in conditions of non-experimental speech production. We investigated whether such units corresponding to single, coherent chunks of speech with syntactically-defined borders, are useful to unravel the neuronal mechanisms underlying real-world human cognition. To this aim, we employed simultaneous electrocorticography (ECoG) and video recordings obtained in pre-neurosurgical diagnostics of epilepsy patients. We transcribed non-experimental, daily hospital conversations, identified IUs in transcriptions of the patients' speech, classified the obtained IUs according to a previously-proposed taxonomy focusing on memory content, and investigated the underlying neuronal activity. In each of our three subjects, we were able to collect a large number of IUs which could be assigned to different functional IU subclasses with a high inter-rater agreement. Robust IU-onset-related changes in spectral magnitude could be observed in high gamma frequencies (70–150 Hz) on the inferior lateral convexity and in the superior temporal cortex regardless of the IU content. A comparison of the topography of these responses with mouth motor and speech areas identified by electrocortical stimulation showed that IUs might be of use for extraoperative mapping of eloquent cortex (average sensitivity: 44.4%, average specificity: 91.1%). High gamma responses specific to memory-related IU subclasses were observed in the inferior parietal and prefrontal regions. IU-based analysis of ECoG recordings during non-experimental communication thus elicits topographically- and functionally-specific effects. We conclude that segmentation of spontaneous real-world speech in linguistically-motivated units is a promising strategy for elucidating the neuronal basis of mental processing during non-experimental communication.

Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication

“…Brain activity during spontaneous, non-experimental communication, however, is a largely unexplored phenomenon, which has been referred to as 'the dark matter' of cognitive neuroscience [40]. To bridge this gap, several labs worldwide have started investigating the neuronal activity underlying language in extraoperatively recorded ECoG in increasingly naturalistic experiments [3,4,[41][42][43][44] as well as in conditions of non-experimental, real-world communication [18,20,21,23,[45][46][47][48][49]. Their reported effects align with previous experimental findings (cf.…”

“…These multimodal recordings have been obtained at the University Medical Center Freiburg for the purpose of pre-neurosurgical diagnostics of epilepsy, and they were donated for research by consented neurological patients. All speakers included in the corpus (data sets from a total amount of eight subjects at different stages of annotation) gave written informed consent that these multimodal recordings would be made available for scientific investigation, and the Ethics Committee of the University Medical Center Freiburg approved the recruitment procedure [20]. The linguistic data in our corpus consist of continuous transcriptions of the subjects' speech production based on the audiovisual signals.…”

“…Phonological properties of the spoken language material, such as the place of phoneme articulation [4,35,36], the manner of articulation, the voicing, and the distinction between vowels and consonants [4], have been studied. Using a syntactically informed approach to segmentation of continuous speech, Derix et al [20] investigated the neuronal differences between ECoG recordings obtained during the production of simple sentences with compared to without memory content. In a sociolinguistic perspective on speech-ECoG data, Derix et al [18] studied dialog partner-specific differences in the neuronal activity during spontaneous communication.…”

“…As in our previous studies [18,20,21], spectral magnitude changes were calculated based on common average reference (CAR)-filtered data using a fast Fourier transformation. Grid electrodes lying within the seizure onset area, identified by epileptologists in ongoing ECoG recordings (C2, D6, E3 in Figure 1(A)), were excluded from all analyses to reduce the impact of strong epileptic spiking activity at these electrodes on our observations.…”

“…Also, a researcher may often have to wait for a suitable patient, in whom the main areas of interest are sufficiently covered by electrodes and whose clinical picture allows for studying the areas of interest (e.g. one may want to make sure the seizure onset zone in patients with epilepsy lies outside the respective areas) [18,20,21]. Important advantages of using this method for speech-related research are that neuronal activity can be recorded directly from the cortical surface, and that recordings, which are usually obtained over an extended period of one to three weeks of pre-neurosurgical diagnostics, can be used to study naturalistic, self-initiated human behavior [21,22] and real-world communication [18,20,23].…”