CluBERT: A Cluster-Based Approach for Learning Sense Distributions in Multiple Languages

Pasini, Tommaso; Scozzafava, Federico; Scarlini, Bianca

doi:10.18653/v1/2020.acl-main.369

Cited by 14 publications

(3 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This architecture was originally proposed for machine translation following the sequence‐to‐sequence paradigm. Later works successfully applied Transformer‐based models to various Natural Language Processing tasks [ZHLL20, PSS20], typically adopting the pretrain‐then‐fine‐tune paradigm [DCLT19]. Its flexibility has been then proven with success also in other domains, such as Computer Vision [DBK*21], Computer Graphics [LYZ22], Speech Processing [WML*20], Reinforcement Learning [CLR*21], and on mixed modalities, such as Vision and Language [KSK21].…”

Section: Related Workmentioning

confidence: 99%

Attention And Positional Encoding Are (Almost) All You Need For Shape Matching

Raganato

Pasi

Melzi

2023

Computer Graphics Forum

View full text Add to dashboard Cite

The fast development of novel approaches derived from the Transformers architecture has led to outstanding performance in different scenarios, from Natural Language Processing to Computer Vision. Recently, they achieved impressive results even in the challenging task of non‐rigid shape matching. However, little is known about the capability of the Transformer‐encoder architecture for the shape matching task, and its performances still remained largely unexplored. In this paper, we step back and investigate the contribution made by the Transformer‐encoder architecture compared to its more recent alternatives, focusing on why and how it works on this specific task. Thanks to the versatility of our implementation, we can harness the bi‐directional structure of the correspondence problem, making it more interpretable. Furthermore, we prove that positional encodings are essential for processing unordered point clouds. Through a comprehensive set of experiments, we find that attention and positional encoding are (almost) all you need for shape matching. The simple Transformer‐encoder architecture, coupled with relative position encoding in the attention mechanism, is able to obtain strong improvements, reaching the current state‐of‐the‐art.

show abstract

Section: Related Workmentioning

confidence: 99%

Attention And Positional Encoding Are (Almost) All You Need For Shape Matching

Raganato

Pasi

Melzi

2023

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…However, their main drawback is that they have difficulty scaling over different languages, since no manually-curated training data is available to them. Automatic methods to produce sense distributions (Pasini, Scozzafava, and Scarlini 2020) or sense-annotated data in languages other than English (Delli Bovi et al 2017;Scarlini, Pasini, and Navigli 2019;Pasini and Navigli 2020; Pasini 1 http://wordnetcode.princeton.edu/glosstag.shtml 2020) have mitigated this limitation, thus allowing supervised approaches to be trained on different languages and to enter a field that was mainly dominated by knowledge-based methods.…”

Section: Related Workmentioning

confidence: 99%

XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation

Pasini

Raganato

Navigli

2021

AAAI

Self Cite

View full text Add to dashboard Cite

Transformer-based architectures brought a breeze of change to Word Sense Disambiguation (WSD), improving models' performances by a large margin. The fast development of new approaches has been further encouraged by a well-framed evaluation suite for English, which has allowed their performances to be kept track of and compared fairly. However, other languages have remained largely unexplored, as testing data are available for a few languages only and the evaluation setting is rather matted. In this paper, we untangle this situation by proposing XL-WSD, a cross-lingual evaluation benchmark for the WSD task featuring sense-annotated development and test sets in 18 languages from six different linguistic families, together with language-specific silver training data. We leverage XL-WSD datasets to conduct an extensive evaluation of neural and knowledge-based approaches, including the most recent multilingual language models. Results show that the zero-shot knowledge transfer across languages is a promising research direction within the WSD field, especially when considering low-resourced languages where large pre-trained multilingual models still perform poorly. We make the evaluation suite and the code for performing the experiments available at https://sapienzanlp.github.io/xl-wsd/.

show abstract

“…Hadiwinoto et al (2019); Coenen et al (2019) showed that this technique works well with BERT too. Pasini et al (2020) uses a combination of BERT embeddings and a knowledge-based WSD model to generate word sense distributions, while Giulianelli et al (2020) uses clustering over the embeddings to detect semantic shifts.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

Xypolopoulos¹,

Tixier²,

Vazirgiannis³

2021

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume

View full text Add to dashboard Cite

The number of senses of a given word, or polysemy, is a very subjective notion, which varies widely across annotators and resources. We propose a novel method to estimate polysemy based on simple geometry in the contextual embedding space. Our approach is fully unsupervised and purely data-driven. Through rigorous experiments, we show that our rankings are well correlated, with strong statistical significance, with 6 different rankings derived from famous human-constructed resources such as WordNet, OntoNotes, Oxford, Wikipedia, etc., for 6 different standard metrics. We also visualize and analyze the correlation between the human rankings and make interesting observations. A valuable by-product of our method is the ability to sample, at no extra cost, sentences containing different senses of a given word. Finally, the fully unsupervised nature of our approach makes it applicable to any language. Code and data are publicly available 1 . * Equal contribution. CX handled the data, generated the rankings, and sampled the examples of section 7. AJPT computed the results, plots, and wrote the paper. Both authors participated in the design of the study.

show abstract

CluBERT: A Cluster-Based Approach for Learning Sense Distributions in Multiple Languages

Cited by 14 publications

References 32 publications

Attention And Positional Encoding Are (Almost) All You Need For Shape Matching

Attention And Positional Encoding Are (Almost) All You Need For Shape Matching

XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation

Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings

Contact Info

Product

Resources

About