Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes

Soler, Aina Garí; Apidianaki, Marianna; Allauzen, Alexandre

doi:10.18653/v1/s19-1002

Cited by 9 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A static embedding baseline (FastText) is also provided. BERT and RoBERTa are reported as the best models without external resources in WiC (Pilehvar and Camacho-Collados, 2019) and Usim(Garí Soler et al, 2019); the previous best reported score is 0.693(Neelakantan et al, 2014) for SCWS. (r: uncentered Pearson correlation, ρ: Spearman correlation, acc: Accuracy)…”

mentioning

confidence: 95%

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Liu¹,

McCarthy²,

Korhonen³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

One of the most powerful features of contextualized models is their dynamic embeddings for words in context, leading to state-ofthe-art representations for context-aware lexical semantics. In this paper, we present a post-processing technique that enhances these representations by learning a transformation through static anchors. Our method requires only another pre-trained model and no labeled data is needed. We show consistent improvement in a range of benchmark tasks that test contextual variations of meaning both across different usages of a word and across different words as they are used in context. We demonstrate that while the original contextual representations can be improved by another embedding space from either contextualized or static models, the static embeddings, which have lower computational requirements, provide the most gains.

show abstract

mentioning

confidence: 95%

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Liu¹,

McCarthy²,

Korhonen³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…We cluster each type of representation for w using k-means with a range of k values (2 ≤ k ≤ 10), and retain the k of the clustering with the highest mean SIL. Additionally, since BERT representations' cosine similarity correlates well with usage similarity (Garí Soler et al, 2019), we experiment with Agglomerative Clustering with average linkage directly on the cosine distance matrix obtained with BERT representations (BERT-AGG). For comparison, we also use Agglomerative Clustering on the gold usage similarity scores from Usim, transformed into distances (Gold-AGG).…”

Section: Sense Clusteringmentioning

confidence: 99%

“…Pre-trained LMs have been shown to successfully leverage sense annotated data for disambiguation (Wiedemann et al, 2019;Reif et al, 2019). The interplay between word type and token-level information in the hidden representations of LSTM LMs has also been explored (Aina et al, 2019), as well as the similarity estimates that can be drawn from contextualized representations without directly addressing word meaning (Ethayarajh, 2019). In recent work, Vulić et al (2020) probe BERT representations for lexical semantics, addressing out-of-context word similarity.…”

Section: Introductionmentioning

confidence: 99%

Let’s Play`Mono`-`Poly`: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Soler

Apidianaki

2021

Transactions of the Association for Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words’ polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research.

show abstract

“…The dashed red lines indicate 1.0 context (right) and 1.0 target word bias, indicating the datasets require the modeling of target words alone or context alone. and Jorge, 2019; Huang et al, 2019;Blevins and Zettlemoyer, 2020), WiC (Pilehvar and Camacho-Collados, 2019;Garí Soler et al, 2019) and entity linking (EL) (Wu et al, 2020;Broscheit, 2019).…”

Section: Mcl-wicmentioning

confidence: 99%

Measuring Context-Word Biases in Lexical Semantic Datasets

Liu¹,

McCarthy²,

Korhonen³

2021

Preprint

View full text Add to dashboard Cite

State-of-the-art contextualized models such as BERT use tasks such as WiC and WSD to evaluate their word-in-context representations. This inherently assumes that performance in these tasks reflect how well a model represents the coupled word and context semantics. This study investigates this assumption by presenting the first quantitative analysis (using probing baselines) on the context-word interaction being tested in major contextual lexical semantic tasks. Specifically, based on the probing baseline performance, we propose measures to calculate the degree of context or word biases in a dataset, and plot existing datasets on a continuum. The analysis shows most existing datasets fall into the extreme ends of the continuum (i.e. they are either heavily context-biased or target-wordbiased) while AM 2 ICO and Sense Retrieval show lower overall biases to challenge a model to represent both the context and target words. Our case study on WiC reveals that human subjects do not share models' strong context biases in the dataset (humans found semantic judgments much more difficult when the target word is missing) and models are learning spurious correlations from context alone. This study demonstrates that models are usually not being tested for word-in-context representations as such in these tasks and results are therefore open to misinterpretation. We recommend our framework as sanity check for context and target word biases of future task design and application in lexical semantics.

show abstract

Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes

Cited by 9 publications

References 19 publications

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Let’s Play`Mono`-`Poly`: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Measuring Context-Word Biases in Lexical Semantic Datasets

Contact Info

Product

Resources

About

Word Usage Similarity Estimation with Sentence Representations and Automatic Substitutes

Cited by 9 publications

References 19 publications

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Towards Better Context-aware Lexical Semantics:Adjusting Contextualized Representations through Static Anchors

Let’s PlayMono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Measuring Context-Word Biases in Lexical Semantic Datasets

Contact Info

Product

Resources

About

Let’s Play`Mono`-`Poly`: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses