Human brain activity for machine attention

Muttenthaler, Lukas; Hollenstein, Nora; Barrett, Maria

doi:10.48550/arxiv.2006.05113

Cited by 6 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Toneva and Wehbe 2019) use fMRI scans to interpret and improve BERT (Devlin et al 2018), a well-known transformer. Relatedly, (Muttenthaler, Hollenstein, and Barrett 2020) use EEG features to modify attention weights in an LSTM based model.…”

Section: Language Models and The Brainmentioning

confidence: 99%

Brain2Word: Decoding Brain Activity for Language Generation

Affolter,

Egressy,

Pascual

et al. 2020

Preprint

View full text Add to dashboard Cite

Brain decoding, understood as the process of mapping brain activities to the stimuli that generated them, has been an active research area in the last years. In the case of language stimuli, recent studies have shown that it is possible to decode fMRI scans into an embedding of the word a subject is reading. However, such word embeddings are designed for natural language processing tasks rather than for brain decoding. Therefore, they limit our ability to recover the precise stimulus. In this work, we propose to directly classify an fMRI scan, mapping it to the corresponding word within a fixed vocabulary. Unlike existing work, we evaluate on scans from previously unseen subjects. We argue that this is a more realistic setup and we present a model that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1 and 13.59% Top-5 accuracy in this challenging task, significantly outperforming all the considered competitive baselines. Furthermore, we use the decoded words to guide language generation with the GPT-2 model. This way, we advance the quest for a system that translates brain activities into coherent text.

show abstract

Section: Language Models and The Brainmentioning

confidence: 99%

Brain2Word: Decoding Brain Activity for Language Generation

Affolter,

Egressy,

Pascual

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Brain-inspired computing is inspired by the human brain, which utilizes multiple types of information, such as visual, sound, and tactus, simultaneously to deal with tasks. Through interactions among various neural systems or neurons, the brain is capable of integrating diverse information while focusing on key elements ( Muttenthaler et al, 2020 ). This information processing approach of the brain has inspired the development of neural network-based multidimensional data fusion techniques ( LeCun et al, 2015 ), such as target detection ( Yuan et al, 2023 ), tracking ( Han et al, 2019a , 2022 ), and recognition ( Zeng et al, 2022b ).…”

Section: Introductionmentioning

confidence: 99%

Dual-band polarimetric HRRP recognition via a brain-inspired multi-channel fusion feature extraction network

Yang,

Zhou,

Yuan

et al. 2023

Front. Neurosci.

View full text Add to dashboard Cite

Radar high-resolution range profile (HRRP) provides geometric and structural information of target, which is important for radar automatic target recognition (RATR). However, due to the limited information dimension of HRRP, achieving accurate target recognition is challenging in applications. In recent years, with the rapid development of radar components and signal processing technology, the acquisition and use of target multi-frequency and polarization scattering information has become a significant way to improve target recognition performance. Meanwhile, deep learning inspired by the human brain has shown great promise in pattern recognition applications. In this paper, a Multi-channel Fusion Feature Extraction Network (MFFE-Net) inspired by the human brain is proposed for dual-band polarimetric HRRP, aiming at addressing the challenges faced in HRRP target recognition. In the proposed network, inspired by the human brain’s multi-dimensional information interaction, the similarity and difference features of dual-frequency HRRP are first extracted to realize the interactive fusion of frequency features. Then, inspired by the human brain’s selective attention mechanism, the interactive weights are obtained for multi-polarization features and multi-scale representation, enabling feature aggregation and multi-scale fusion. Finally, inspired by the human brain’s hierarchical learning mechanism, the layer-by-layer feature extraction and fusion with residual connections are designed to enhance the separability of features. Experiments on simulated and measured datasets verify the accurate recognition capability of MFFE-Net, and ablative studies are conducted to confirm the effectiveness of components of network for recognition.

show abstract

“…Although covariate shift has been generally observed in brain signal data due to intra-and inter-subject variability (Lund et al 2005;Saha and Baumert 2020), previous work demonstrated promising transfer learning ability in brain signal decoding using deep learning models (Roy et al 2020;Zhang et al 2020;Makin, Moses, and Chang 2020). Furthermore, various studies (Muttenthaler, Hollenstein, and Barrett 2020;Hollenstein et al 2021Hollenstein et al , 2019Hale et al 2018;Schwartz and Mitchell 2019) have experimented with connecting brain signal decoding to NLP models, by either using brain signals as an additional modality for improving performance on NLP tasks or using NLP models to understand how the human brain encodes language.…”

Section: Introductionmentioning

confidence: 99%

Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification

Wang

2022

AAAI

View full text Add to dashboard Cite

State-of-the-art brain-to-text systems have achieved great success in decoding language directly from brain signals using neural networks. However, current approaches are limited to small closed vocabularies which are far from enough for natural communication. In addition, most of the high-performing approaches require data from invasive devices (e.g., ECoG). In this paper, we extend the problem to open vocabulary Electroencephalography(EEG)-To-Text Sequence-To-Sequence decoding and zero-shot sentence sentiment classification on natural reading tasks. We hypothesis that the human brain functions as a special text encoder and propose a novel framework leveraging pre-trained language models (e.g., BART). Our model achieves a 40.1% BLEU-1 score on EEG-To-Text decoding and a 55.6% F1 score on zero-shot EEG-based ternary sentiment classification, which significantly outperforms supervised baselines. Furthermore, we show that our proposed model can handle data from various subjects and sources, showing great potential for a high-performance open vocabulary brain-to-text system once sufficient data is available. The code is made publicly available for research purpose at https://github.com/MikeWangWZHL/EEG-To-Text.

show abstract

Human brain activity for machine attention

Cited by 6 publications

References 25 publications

Brain2Word: Decoding Brain Activity for Language Generation

Brain2Word: Decoding Brain Activity for Language Generation

Dual-band polarimetric HRRP recognition via a brain-inspired multi-channel fusion feature extraction network

Open Vocabulary Electroencephalography-to-Text Decoding and Zero-Shot Sentiment Classification

Contact Info

Product

Resources

About