Deep Learning Human Mind for Automated Visual Classification

Abstract

What if we could effectively read the mind and transfer human visual capabilities to computer vision methods? In this paper, we aim at addressing this question by developing the first visual object classifier driven by human brain signals. In particular, we employ EEG data evoked by visual object stimuli combined with Recurrent Neural Networks (RNN) to learn a discriminative brain activity manifold of visual categories. Afterward, we train a Convolutional Neural Network (CNN)-based regressor to project images onto the learned manifold, thus effectively allowing machines to employ human brain-based features for automated visual classification. We use a 32-channel EEG to record brain activity of seven subjects while looking at images of 40 ImageNet object classes. The proposed RNN-based approach for discriminating object classes using brain signals reaches an average accuracy of about 40%, which outperforms existing methods attempting to learn EEG visual object representations. As for automated object categorization, our human brain-driven approach obtains competitive performance, comparable to those achieved by powerful CNN models, both on ImageNet and CalTech 101, thus demonstrating its classification and generalization capabilities. This gives us a real hope that, indeed, human mind can be read and transferred to machines.

Cogni-Net: Cognitive Feature Learning Through Deep Visual Perception

“…Furthermore, like images, we want our signal network to be translation invariant. Significantly, stacked BLSTM layers which work surprisingly well under complex conditions [8] help in obtaining a high-level encoding of features from within these raw brain signals. To tackle the problem of varying temporal lengths of the brain signals, we follow the work proposed in [8] and represent every input brain EEG signal as a matrix, M ∈ R 440×128 where the rows represent the effective 440 timesteps and the columns represent the 128 channels [8] at each timestep.…”

“…The batch size of 32 was found to work the best for the conducted experiments. The Dataset For our experiments we use the dataset released in [8]. This visual stimuli dataset is a small subset of the ImageNet dataset with 40 classes with 50 images per class and accompanying EEG signal data.…”

“…Our work attempts to learn the cognitive-activity feature-space by utilizing these very signals (EEG) generated by humans during cognition and use them to discriminate between visual object categories. Works such as [1,2,3,4,5,6,7] show that brain signals evoked by a specific stimuli contain information about image categories, with [8] proving that they can be exploited to discriminate between image classes. Unlike [8], our work attempts to utilize the link between the visual stimuli (images) and cognitive activity (EEG signals), taking inspiration from [9] which attempts to learn acoustic representations from unlabeled videos.…”

“…Unlike [8], our work attempts to utilize the link between the visual stimuli (images) and cognitive activity (EEG signals), taking inspiration from [9] which attempts to learn acoustic representations from unlabeled videos. The authors of [8] used EEG signals for image classification, for which we already have very accurate deep learning models. Furthermore storing images for classification is very costly.…”

Natural image reconstruction from brain waves: a novel visual BCI system with native feedback

“…[14,15] propose a model for natural image reconstruction from fMRI brain signal data recorded from a subject while he observes the original images. There have been similar reports on similar EEG-based reconstructions [16], but the reliability of the studies has met serious controversy [17].…”