Let’s Play<tt>Mono</tt>-<tt>Poly</tt>: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Soler, Aina Garí; Apidianaki, Marianna

doi:10.1162/tacl_a_00400

Cited by 24 publications

(7 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PLMs, unlike embedding methods such as GloVe and word2vec, have been shown to learn a good estimation of the different senses of words during pre-training (Vulić et al 2020;Garí Soler & Apidianaki 2021;Haber & Poesio 2021). In addition, significant gains have been made on the task of word sense disambiguation by the use of PLMs (Loureiro & Jorge 2019;Loureiro et al 2022), further reinforcing this notion.…”

Section: The Emergence Of Construction Grammar In Pre-trained Languag...mentioning

confidence: 95%

CxGBERT: BERT meets Construction Grammar

Madabushi

Romain²,

Divjak³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

While lexico-semantic elements no doubt capture a large amount of linguistic information, it has been argued that they do not capture all information contained in text. This assumption is central to constructionist approaches to language which argue that language consists of constructions, learned pairings of a form and a function or meaning that are either frequent or have a meaning that cannot be predicted from its component parts. BERT's training objectives give it access to a tremendous amount of lexico-semantic information, and while BERTology has shown that BERT captures certain important linguistic dimensions, there have been no studies exploring the extent to which BERT might have access to constructional information. In this work we design several probes and conduct extensive experiments to answer this question. Our results allow us to conclude that BERT does indeed have access to a significant amount of information, much of which linguists typically call constructional information. The impact of this observation is potentially far-reaching as it provides insights into what deep learning methods learn from text, while also showing that information contained in constructions is redundantly encoded in lexicosemantics.

show abstract

Section: The Emergence Of Construction Grammar In Pre-trained Languag...mentioning

confidence: 95%

CxGBERT: BERT meets Construction Grammar

Madabushi

Romain²,

Divjak³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…However, the results are not straightforward. Whereas some research found that polysemy detection, differentiation between polysemous words regarding NoS (Yenicelik et al, 2020), and even sense disambiguation is possible within transformer models (Soler & Apidianaki, 2021), some claim that structures that are present in these pre-trained models are so seamless that information extraction is impossible (Wiedemann et al, 2019). The issue with these models is that the more advanced and better at disambiguation, the more technically complex they are.…”

Section: Describing Sensesmentioning

confidence: 99%

An Open Database of Senses for Serbian Polysemous Nouns, Verbs, and Adjectives

Mišić,

Anđelić,

Ilić

et al. 2024

SLM

View full text Add to dashboard Cite

The goal of this study was twofold. On the one hand, we aimed to obtain the number of senses estimations for polysemous nouns, verbs, and adjectives, as well as frequencies of occurrence for each sense based on native speakers’ intuitions. From that, we derived multiple quantifications to describe each word. We determined what the senses of each word are by allowing participants to list everything they recall a word could denote. We categorised the responses and counted the senses per word. On the other hand, we aimed to investigate the reliability of native speakers’ intuitions. The novelty in our approach was that multiple researchers who categorised the responses were also treated as participants in the study. In this paper, we present the database and explore the multiple-coder approach. This process generated data on various levels that could be useful for psycholinguistic research. We share both qualitative data – participants’ individual responses and classification into senses by our coders, and quantitative data – number of senses counts, sense probabilities, entropy, redundancy, familiarity, and concreteness measures. All data are freely available at https://osf.io/ukcrg/.

show abstract

“…These contextualized embeddings have been shown to improve performance on a number of downstream Natural Language Processing tasks involving lexical ambiguity, such as word sense disambiguation (Aina et al, 2019; Loureiro et al, 2020). Past work also suggests that BERT can be used to distinguish monosemous and polysemous words, or even polysemy and homonymy (Haber & Poesio, 2020a, 2020b; Nair et al, 2020; Soler & Apidianaki, 2021), and that BERT’s representations encode sense-like information (Karidi et al, 2021). Most relevantly for our purposes, BERT’s contextualized embeddings are well-suited for measuring contextual distance in a graded manner––given two contextualized embeddings of an ambiguous target word (e.g., for “marinated lamb ” and “friendly lamb ”), we can compute the cosine distance between those vectors, a metric often used to assess proximity in vector space 4 .…”

Section: Current Workmentioning

confidence: 99%

Word meaning is both categorical and continuous.

Trott¹,

Bergen²

2023

Psychological Review

View full text Add to dashboard Cite

Most words have multiple meanings, but there are foundationally distinct accounts for this. Categorical theories posit that humans maintain discrete entries for distinct word meanings, as in a dictionary. Continuous ones eschew discrete sense representations, arguing that word meanings are best characterized as trajectories through a continuous state space. Both kinds of approach face empirical challenges. In response, we introduce two novel "hybrid" theories, which reconcile discrete sense representations with a continuous view of word meaning. We then report on two behavioral experiments, pairing them with an analytical approach relying on neural language models to test these competing accounts. The experimental results are best explained by one of the novel hybrid accounts, which posits both distinct sense representations and a continuous meaning space. This hybrid account accommodates both the dynamic, context-dependent nature of word meaning, as well as the behavioral evidence for category-like structure in human lexical knowledge. We further develop and quantify the predictive power of several computational implementations of this hybrid account. These results raise questions for future research on lexical ambiguity, such as why and when discrete sense representations might emerge in the first place. They also connect to more general questions about the role of discrete versus gradient representations in cognitive processes and suggest that at least in this case, the best explanation is one that integrates both factors: Word meaning is both categorical and continuous.

show abstract

Let’s Play`Mono`-`Poly`: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Cited by 24 publications

References 45 publications

CxGBERT: BERT meets Construction Grammar

CxGBERT: BERT meets Construction Grammar

An Open Database of Senses for Serbian Polysemous Nouns, Verbs, and Adjectives

Word meaning is both categorical and continuous.

Contact Info

Product

Resources

About

Let’s PlayMono-Poly: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses

Cited by 24 publications

References 45 publications

CxGBERT: BERT meets Construction Grammar

CxGBERT: BERT meets Construction Grammar

An Open Database of Senses for Serbian Polysemous Nouns, Verbs, and Adjectives

Word meaning is both categorical and continuous.

Contact Info

Product

Resources

About

Let’s Play`Mono`-`Poly`: BERT Can Reveal Words’ Polysemy Level and Partitionability into Senses