Music can be reconstructed from human auditory cortex activity using nonlinear decoding models

Abstract

Music is core to human experience, yet the precise neural dynamics underlying music perception remain unknown. We analyzed a unique intracranial electroencephalography (iEEG) dataset of 29 patients who listened to a Pink Floyd song and applied a stimulus reconstruction approach previously used in the speech domain. We successfully reconstructed a recognizable song from direct neural recordings and quantified the impact of different factors on decoding accuracy. Combining encoding and decoding analyses, we found a right-hemisphere dominance for music perception with a primary role of the superior temporal gyrus (STG), evidenced a new STG subregion tuned to musical rhythm, and defined an anterior–posterior STG organization exhibiting sustained and onset responses to musical elements. Our findings show the feasibility of applying predictive modeling on short datasets acquired in single patients, paving the way for adding musical elements to brain–computer interface (BCI) applications.

How to deal with mind-reading technologies

“…The other decodes the brain activity that correlates with the manifestation of thoughts (Tang et al, 2023 ). Thanks to expert software trained on the individual who is the subject of the experiment, it is also possible to reconstruct verbalizations, images and even the music being heard by the subject (Bellier et al, 2023 ).…”

Embracing digital innovation in neuroscience: 2023 in review at NEUROCCINO

“…Decoding individual brain signals, often referred to as ‘mind reading’ in the lay media, was also exemplified by the ‘Pink Floyd study’, where neural activity was recorded from 29 people with epilepsy with intracranial electrodes who listened to Another Brick in the Wall (Bellier et al 2023 ). Using independent component analysis (ICA), meaningful components of the neural activity related to the music, such as vocals and instruments, were extracted.…”

“…Using independent component analysis (ICA), meaningful components of the neural activity related to the music, such as vocals and instruments, were extracted. The discussion highlighted the need for well-defined data pools, the lack of generalisability given the invasive nature of the study, and the lack of control for song familiarity (9 × 23fk_fHoM; Bellier et al 2023 ). Another study also delved into the details of Direct Electrical Stimulation Mapping (DES) in neurosurgery, highlighting its invasive yet crucial role in localising functional boundaries around tumours to prevent harm to the language system (leEddoDBihk; Collée et al 2023 ).…”

Functional Near-Infrared Spectrometry as a Useful Diagnostic Tool for Understanding the Visual System: A Review

“…Remarkably, the scope of reconstruction extends beyond images to include auditory stimuli. Utilizing non-linear decoding models applied to intracranial electroencephalography (iEEG) data obtained from 29 epilepsy patients, Bellier et al successfully reconstructed Pink Floyd’s ‘Another Brick in the Wall’ from individuals who were passively listening to the track [ 47 ]. Similarly, Denk et al accomplished music reconstruction using functional magnetic resonance imaging (fMRI) data [ 48 ].…”