Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

Xypolopoulos, Christos; Tixier, Antoine J.‐P.; Vazirgiannis, Michalis

doi:10.18653/v1/2021.eacl-main.297

Cited by 3 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Goodwin et al, 2020;Pimentel et al, 2020b,a;Aghazadeh et al, 2022;Tucker et al, 2022;Arps et al, 2022). In contrast, descriptive probing looks into the intrinsic structure of representations Zhou and Srikumar, 2021;Xypolopoulos et al, 2021;Chang et al, 2022), and focuses on discovering properties of the representation using cluster analysis (Aharoni and Goldberg, 2020) or visualization technqiues Vig, 2019).…”

Section: Two Characteristics Of Representationsmentioning

confidence: 99%

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

2023

View full text Add to dashboard Cite

As researchers achieve unprecedented technological breakthroughs in natural language processing, the need to understand the systems underlying these advances is more pertinent than ever. BlackboxNLP, now in its sixth iteration, has played an important role in bringing together scholars from a diverse range of backgrounds in order to rigorously study the behavior, representations, and computations of "black-box" neural network models. Our workshop showcases original, cutting-edge research on topics including but not limited to:• analysis of representations via probing and related techniques;• explanation methods such as feature attribution, free-text explanations, or structured explanations; • interdisciplinary methods (e.g., from neuroscience, cognitive science, computer vision, etc.); • interpretable architectures and neural network modules;• mechanistic interpretability and reverse engineering of neural computations; • open-source tools for analysis, visualization, and/or explanation; • opinions about the state of interpretability and explainable NLP; and • targeted evaluations using simplified or formal languages.The sixth BlackboxNLP workshop will be held in Singapore on December 7, 2023, hosted by the Conference on Empirical Methods in Natural Language Processing (EMNLP). 29 full papers and 17 non-archival extended abstracts were accepted for in-person and online presentations, from a total of 66 submissions. This year's workshop will also feature 19 papers on interpretability from the Findings of the ACL: EMNLP 2023, as well as two invited talks and a panel discussion with experts in the field.BlackboxNLP 2023 would not have been possible without the high-quality peer reviews submitted by our program committee, as well as the logistical assistance provided by the EMNLP organizing committee. We gratefully acknowledge financial support from our sponsors, Google and Apple. Our invited speakers, panelists, authors, and presenters have allowed us to put together an outstanding program for all participants to enjoy.Welcome to BlackboxNLP! We look forward to seeing you in Singapore and online.

show abstract

Section: Two Characteristics Of Representationsmentioning

confidence: 99%

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

2023

View full text Add to dashboard Cite

show abstract

“…Conneau et al, 2018;Kassner and Schütze, 2020;Goodwin et al, 2020;Pimentel et al, 2020b,a;Aghazadeh et al, 2022;Tucker et al, 2022;Gonen et al, 2022;Arps et al, 2022). In contrast, descriptive probing looks into the intrinsic structure of representations (Ethayarajh, 2019;Zhou and Srikumar, 2021;Xypolopoulos et al, 2021;Chang et al, 2022), and focuses on discovering properties of the representation using cluster analysis (Aharoni and Goldberg, 2020) or visualization technqiues (Reimers et al, 2019;Vig, 2019).…”

Section: Two Characteristics Of Representationsmentioning

confidence: 99%

METAPROBE: A Representation- and Task-Agnostic Probe

Zhou,

Srikumar

2023

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

Probing contextualized representations typically involves comparing task-specific model predictions against ground truth linguistic labels. Although this methodology shows what information can be recovered by a classifier, it does not reveal how a classifier uses the representation to make its decision. To address the latter problem, we ask: Do task-classifiers rely on representation-and task-independent geometric patterns in the embedding space? We explore this question by developing METAPROBE, an approach that uses geometric properties of representations to predict the behavior of taskspecific classifiers (i.e, their predictions as opposed to the ground truth). Our experiments reveal the existence of universal geometric patterns across representations that can predict classifier predictions. Consequently, this allows us to posit a geometric explanation for the impressive performance of contextualized representations.

show abstract

“…Understanding the geometry of the BERT-space is not easy. Some attempts in this direction have been made (Coenen et al, 2019;Ethayarajh, 2019;Michael et al, 2020;Mickus et al, 2020;Xypolopoulos et al, 2021;Garí Soler and Apidianaki, 2020), but a more thorough investigation is lacking. As opposed to predictive methods such as probing, descriptive methods that rely on geometric features of the space analyze the information in CRs directly.…”

Section: Analyzing Contextual Representationsmentioning

confidence: 99%

Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords

Karidi¹,

Zhou²,

Schneider³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses. By inducing a contextualized "pseudoword" as a stand-in for a static embedding in the input layer, and then performing masked prediction of a word in the sentence, we are able to investigate the geometry of the BERT-space in a controlled manner around individual instances. Using our method on a set of carefully constructed sentences targeting ambiguous English words, we find substantial regularity in the contextualized space, with regions that correspond to distinct word senses; but between these regions there are occasionally "sense voids"-regions that do not correspond to any intelligible sense. 1

show abstract

Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

Cited by 3 publications

References 28 publications

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP

METAPROBE: A Representation- and Task-Agnostic Probe

Putting Words in BERT’s Mouth: Navigating Contextualized Vector Spaces with Pseudowords

Contact Info

Product

Resources

About