EEG2Mel: Reconstructing Sound from Brain Responses to Music

Abstract

Information retrieval from brain responses to auditory and visual stimuli has shown success through classification of song names and image classes presented to participants while recording EEG signals. Information retrieval in the form of reconstructing auditory stimuli has also shown some success, but here we improve on previous methods by reconstructing music stimuli well enough to be perceived and identified independently. Furthermore, deep learning models were trained on time-aligned music stimuli spectrum for each corresponding one-second window of EEG recording, which greatly reduces feature extraction steps needed when compared to prior studies. The NMED-Tempo and NMED-Hindi datasets of participants passively listening to full length songs were used to train and validate Convolutional Neural Network (CNN) regressors. The efficacy of raw voltage versus power spectrum inputs and linear versus mel spectrogram outputs were tested, and all inputs and outputs were converted into 2D images. The quality of reconstructed spectrograms was assessed by training classifiers which showed 81% accuracy for mel-spectrograms and 72% for linear spectrograms (10% chance accuracy). Lastly, reconstructions of auditory music stimuli were discriminated by listeners at an 85% success rate (50% chance) in a two-alternative matchto-sample task.

Nested Dynamical Modeling of Neural Systems: A strategy and some consequences

“…Here, recorded brain activity are used to make predictions about features in the world and have increasingly been integrated with machine learning tools, like support vector machines (Glaser et al, 2020). Illustrative cases of decoding from large data sets of neural recordings include such machine learning-based analyses of electroencephalography (EEG) data as those that differentiate phonetic prototypes from ambiguous speech sounds (Mahmud et al, 2021), decoding visual percepts during binocular rivalry experiments (Krisst & Luck, 2022), and reconstructing musical stimuli (Ramirez-Aristizabal & Kello, 2022). ), to name a few.…”

On multiscale analyses of neural processing, motor movement, and cognition

“…Cognitive psychology tends to focus at the spatial scales of brains and cortical regions (though some psychologists have studied transactive memory; Wegner, 1987). By contrast, neuroscience research spans a vast range from the nanoscopic scale of neurotransmitter molecules (Poznanski et al, 2022a) to the microscopic scale of neurons (Hubel & Weisel, 1959) to the mesoscopic scale of groups of neurons coordinating their activity (Freeman, 2000) to the macroscopic scale of entire brains (Ramirez-Aristizabal & Kello, 2022) and even further to coordinated activity among far too coarse-grained of an of analysis. Likewise, a molecular neuroscientist who studies the action of ion pumps in a cell membrane might find the action potentials recorded by the electrophysiologist to be too coarse-grained a method as well.…”