Neural decoding of music from the EEG

Abstract

Neural decoding models can be used to decode neural representations of visual, acoustic, or semantic information. Recent studies have demonstrated neural decoders that are able to decode accoustic information from a variety of neural signal types including electrocortiography (ECoG) and the electroencephalogram (EEG). In this study we explore how functional magnetic resonance imaging (fMRI) can be combined with EEG to develop an accoustic decoder. Specifically, we first used a joint EEG-fMRI paradigm to record brain activity while participants listened to music. We then used fMRI-informed EEG source localisation and a bi-directional long-term short term deep learning network to first extract neural information from the EEG related to music listening and then to decode and reconstruct the individual pieces of music an individual was listening to. We further validated our decoding model by evaluating its performance on a separate dataset of EEG-only recordings. We were able to reconstruct music, via our fMRI-informed EEG source analysis approach, with a mean rank accuracy of 71.8% ($$n~=~18$$ n = 18 , $$p~<~0.05$$ p < 0.05 ). Using only EEG data, without participant specific fMRI-informed source analysis, we were able to identify the music a participant was listening to with a mean rank accuracy of 59.2% ($$n~=~19$$ n = 19 , $$p~<~0.05$$ p < 0.05 ). This demonstrates that our decoding model may use fMRI-informed source analysis to aid EEG based decoding and reconstruction of acoustic information from brain activity and makes a step towards building EEG-based neural decoders for other complex information domains such as other acoustic, visual, or semantic information.

Music-evoked emotions classification using vision transformer in EEG signals

“…This was supported by the observed augmentation of spectral frequency power in the beta range throughout the entire brain, as well as in the theta range within the parietal region, and in the gamma range across the entire brain. In their study, Daly ( 2023 ) investigated the integration of functional magnetic resonance imaging (fMRI) and EEG techniques to develop an acoustic decoder for the purpose of classifying music emotions. The study employed an EEG-fMRI combined paradigm to capture neural responses during music listening among individuals.…”

Exploring Brain Dynamics via EEG and Steady-State Activation Map Networks in Music Composition

“…This approach not only expands the sample size but also allows deep learning algorithms to extract richer information, achieving commendable results in emotion recognition studies. However, the suitability of this method for more complex cognitive activities [ 43 , 44 ], such as those involved in musical creativity, requires further experimental validation. The present study selected individuals with the highest recognition rates from two groups of subjects.…”

Covert cortical processing: a diagnosis in search of a definition

“…comatose, vegetative state (VS)/unresponsive wakefulness syndrome (UWS), or minimally conscious state (MCS)) and capacity for recovery, but increasingly are beginning to be used to query the content of a patient’s consciousness, including imagined speech and scenes, in the absence of overt interaction ( Scotti et al . , Sorger and Goebel 2020 , Daly 2023 , Giraud and Su 2023 , Ozcelik and VanRullen 2023 , Tang et al. 2023 , Willett et al.…”