No abstract

No abstract

Decoding how the brain represents perceived as well as imagined stimuli is a fundamental challenge in neuroscience that has attracted attention over decades. For visual stimuli, not only image classification but also image reconstruction (i.e., synthesis) has been demonstrated. Namely, blood-oxygen-level-dependent (BOLD) signals from the visual cortex acquired via functional magnetic resonance imaging (fMRI) have enabled the computational synthetization of images that were visually observed [1] or imagined during sleep. [2] In the case of auditory stimuli, sound synthesis from brain activity patterns is rendered more complicated by the requirement of high temporal resolution for discerning the rapid succession of perceptual entities, such as phonemes. Yet, the successful synthesis of presented and recalled or imagined sounds, and in particular speech, would be of considerable interest in both unveiling the coding mechanisms that ultimately support language, and eventually providing a means of communication that bypasses vocalization as a brain-computer interface. Previous studies have relied upon electrocorticography (ECoG) to obtain recordings directly from the auditory cortical areas and thereby attempt to synthesize speech sounds. [3-5] Despite recent advances in machine learning such as the application of transfer learning and recurrent neural networks (RNNs), the synthesis of intelligible speech even using ECoG remains challenging. Moreover, the invasive nature of this recording method fundamentally curtails its deployability to the general population and intrinsically limits the coverage area to that reachable via a clinically motivated craniotomy