Blackbox Meets Blackbox: Representational Similarity &amp; Stability Analysis of Neural Language Models and Brains

Abnar, Samira; Beinborn, Lisa; Choenni, R.; Zuidema, Willem

doi:10.18653/v1/w19-4820

Cited by 53 publications

(44 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Voita et al (2019a) used a form of canonical correlation analysis (PW-CCA; Morcos et al, 2018) to study the layerwise evolution of representations, while Saphra and Lopez (2019) explored how these representations evolve during training. Abnar et al (2019) used Representational Similarity Analysis (RSA; Laakso and Cottrell, 2000;Kriegeskorte et al, 2008) to study the effect of context on encoder representations, while Chrupała and Alishahi (2019) correlated them with syntax.…”

Section: Related Workmentioning

confidence: 99%

“…RSA is a technique for measuring the similarity between two different representation spaces for a given set of stimuli. Originally developed for neuroscience (Kriegeskorte et al, 2008), it has become increasingly used to analyze similarity between neural network activations (Abnar et al, 2019;Chrupała and Alishahi, 2019). The method works by using a common set of n examples, used to create two sets of representations.…”

Section: Representational Similarity Analysismentioning

confidence: 99%

“…Base vs. a fine-tuned model). Following previous applications of RSA to text representations (Abnar et al, 2019;Chrupała and Alishahi, 2019), we adopt the cosine similarity kernel.…”

Section: Representational Similarity Analysismentioning

confidence: 99%

See 2 more Smart Citations

What Happens To BERT Embeddings During Fine-tuning?

Merchant¹,

Rahimtoroghi²,

Pavlick³

et al. 2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

While much recent work has examined how linguistic information is encoded in pretrained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks.Using a suite of analysis techniques-supervised probing, unsupervised similarity analysis, and layer-based ablations-we investigate how fine-tuning affects the representations of the BERT model. We find that while fine-tuning necessarily makes some significant changes, there is no catastrophic forgetting of linguistic phenomena. We instead find that fine-tuning is a conservative process that primarily affects the top layers of BERT, albeit with noteworthy variation across tasks. In particular, dependency parsing reconfigures most of the model, whereas SQuAD and MNLI involve much shallower processing. Finally, we also find that fine-tuning has a weaker effect on representations of out-of-domain sentences, suggesting room for improvement in model generalization.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Representational Similarity Analysismentioning

confidence: 99%

See 1 more Smart Citation

What Happens To BERT Embeddings During Fine-tuning?

Merchant¹,

Rahimtoroghi²,

Pavlick³

et al. 2020

Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

show abstract

“…Huth et al (2016) replicated and extended these results using distributed word representations, and Pereira et al (2018) extended these results to sentence stimuli. Wehbe et al (2014), Qian et al (2016, Jain and Huth (2018), and Abnar et al (2019) next introduced more complex word and sentence meaning representations, demonstrating that neural network language models could better account for brain activation by incorporating repre-Figure 1: Brain decoding methodology. We use human brain activations in response to sentences to predict how neural networks represent those same sentences.…”

Section: Related Workmentioning

confidence: 99%

Linking artificial and human neural representations of language

Gauthier¹,

Lévy²

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

What information from an act of sentence understanding is robustly represented in the human brain? We investigate this question by comparing sentence encoding models on a brain decoding task, where the sentence that an experimental participant has seen must be predicted from the fMRI signal evoked by the sentence. We take a pre-trained BERT architecture as a baseline sentence encoding model and fine-tune it on a variety of natural language understanding (NLU) tasks, asking which lead to improvements in brain-decoding performance.We find that none of the sentence encoding tasks tested yield significant increases in brain decoding performance. Through further task ablations and representational analyses, we find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Our results constrain the space of NLU models that could best account for human neural representations of language, but also suggest limits on the possibility of decoding fine-grained syntactic information from fMRI human neuroimaging.

show abstract

“…Moreover, we use RSA to compare brain imaging data with computational models, and it is worth investigating applications of RSA beyond brain imaging data. For example, recent work has used RSA to compare representational spaces across computational language models and their individual components (Gauthier and Levy, 2019;Abnar et al, 2019;Chrupała and Alishahi, 2019). We also feel that the multi-arrangement method is underutilized in NLP for the shortcomings it addresses in existing semantic judgement acquisition techniques.…”

Section: Resultsmentioning

confidence: 99%

Understanding and Improving Word Embeddings through a Neuroscientific Lens

Fereidooni

Mocz

Radev

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite the success of models making use of word embeddings on many natural language tasks, these models often perform significantly worse than humans on several natural language understanding tasks. This difference in performance motivates us to ask: (1) if existing word vector representations have any basis in the brain’s representational structure for individual words, and (2) whether features from the brain can be used to improve word embedding model performance, defined as their correlation with human semantic judgements. To answer the first question, we compare the representational spaces of existing word embedding models with that of brain imaging data through representational similarity analysis. We answer the second question by using regression-based learning to constrain word vectors to the features of the brain imaging data, thereby determining if these modified word vectors exhibit increased performance over their unmodified counterparts. To collect semantic judgements as a measure of performance, we employed a novel multi-arrangement method. Our results show that there is variance in the representational space of the brain imaging data that remains uncaptured by word embedding models, and that brain imaging data can be used to increase their coherence with human performance.

show abstract

Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains

Cited by 53 publications

References 30 publications

What Happens To BERT Embeddings During Fine-tuning?

What Happens To BERT Embeddings During Fine-tuning?

Linking artificial and human neural representations of language

Understanding and Improving Word Embeddings through a Neuroscientific Lens

Contact Info

Product

Resources

About