Image-based eeg classification of brain responses to song recordings

Abstract

Classifying EEG responses to naturalistic acoustic stimuli is of theoretical and practical importance, but standard approaches are limited by processing individual channels separately on very short sound segments (a few seconds or less). Recent developments have shown classification for music stimuli (∼ 2 mins) by extracting spectral components from EEG and using convolutional neural networks (CNNs). This paper proposes an efficient method to map raw EEG signals to individual songs listened for end-to-end classification. EEG channels are treated as a dimension of a [Channel × Sample] image tile, and images are classified using CNNs. Our experimental results ( 88.7%) compete with state-of-the-art methods (85.0%), yet our classification task is more challenging by processing longer stimuli that were similar to each other in perceptual quality, and were unfamiliar to participants. We also adopt a transfer learning scheme using a pre-trained ResNet-50, confirming the effectiveness of transfer learning despite image domains being unrelated from each other.

EEG2Mel: Reconstructing Sound from Brain Responses to Music

“…Each arrow in the figure is a model showing the designated input to target mapping. Prior classification studies have shown a preference for input representations either being raw voltage or a Power Spectral Density (PSD), and their ability to boost performance in CNNs [5,6]. As a regression task, we also find it important to compare target representations since linear and melspectrogram representations have tradeoffs.…”

“…Table 1 shows a summary of how the model architecture is constructed; the models contain five convolutional layers with the last convolutional layer being the first layer that reduces the dimensionality of the input. A Max Pooling layer with a small pool size is chosen over a Global Average Pooling layer used previously during EEG classification [6] using the same dataset, because it was necessary to limit shrinkage since the output layer was the size of the spectral target.…”

“…Non-monotonic activation functions Swish and Mish were tried due to their success in improving image processing in deep networks, but they did not present any evident advantage over ReLu during training. Lastly, we use Adaptive Moment Estimation (Adam) as an optimizer with a 0.0015 learning rate and initialized weights with a He uniform distribution which have shown advantages in similar training procedures [6,12].…”

“…This type of validation has been used before in the image stimuli domain when trying to objectively test the quality of reconstructed image classes from generative models [7]. The classifiers were CNN based following the architecture from a previous study where the EEG was classified into song name classes [6]. Table 2 demonstrates a summary of the results from the classified outputs across modeling approaches.…”

“…The researchers trained models with the power spectrum density EEG representations and compared it to the raw EEG voltage representations. Training on the raw input representation was able to show that not only was end-to-end classification possible, but that it could also achieve State-of-the-Art performance at 88.6% with 10% chance on unfamiliar music [6]. Studies moving beyond classification approaches and into trying to reconstruct the stimuli have seen some initial success with the use of generative models and careful feature extraction.…”