XL-WSD: An Extra-Large and Cross-Lingual Evaluation Framework for Word Sense Disambiguation

Abstract

Transformer-based architectures brought a breeze of change to Word Sense Disambiguation (WSD), improving models' performances by a large margin. The fast development of new approaches has been further encouraged by a well-framed evaluation suite for English, which has allowed their performances to be kept track of and compared fairly. However, other languages have remained largely unexplored, as testing data are available for a few languages only and the evaluation setting is rather matted. In this paper, we untangle this situation by proposing XL-WSD, a cross-lingual evaluation benchmark for the WSD task featuring sense-annotated development and test sets in 18 languages from six different linguistic families, together with language-specific silver training data. We leverage XL-WSD datasets to conduct an extensive evaluation of neural and knowledge-based approaches, including the most recent multilingual language models. Results show that the zero-shot knowledge transfer across languages is a promising research direction within the WSD field, especially when considering low-resourced languages where large pre-trained multilingual models still perform poorly. We make the evaluation suite and the code for performing the experiments available at https://sapienzanlp.github.io/xl-wsd/.

Quantum Interference Model for Semantic Biases of Glosses in Word Sense Disambiguation

“…We perform experiments under the cross-lingual datasets to answer Q3. The datasets 7 are proposed by Pasini et al (Pasini, Raganato, and Navigli 2021) Since it is difficult to obtain glosses in the corresponding language, here we adopt a compromise solution to replace glosses in other languages with English glosses. The pretrained language model used in our model is also adjusted to bert-base-multilingualcased accordingly.…”

UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation

“…Knowledgebased methods depend on other sources of linguistic knowledge (Wang and Wang, 2020). In general, knowledge-based methods are outperformed by contemporary supervised methods (Pasini et al, 2021). Today, state-of-the-art WSD systems approach accuracy limits imposed by inter-annotator agreement (Maru et al, 2022).…”

Code-Switching with Word Senses for Pretraining in Neural Machine Translation

“…To measure the impact of disambiguation quality on the final translation performance, we adopt two WSD systems, namely, ESCHER, a SOTA open-inventory WSD model, and AMuSE-WSD (Orlando et al, 2021), a faster but still competitive, off-the-shelf classification system. We note that, since ESCHER was originally only released for English, we also trained models for other languages on the multilingual datasets in XL-WSD (Pasini et al, 2021). We provide further training details in Appendix A.2.…”

“…For this reason, we train three distinct ESCHER models specifically tailored for Italian, Spanish, and French. We adopt the methodology outlined in the original work but adapt it to train the models using the XL-WSD (Pasini et al, 2021) splits for the respective languages and use mDeBERTa (He et al, 2023) as the underlying multilingual transformer architecture. By doing so, we ensure compatibility and improve performance in the targeted linguistic contexts as reported in Barba et al (2021b) without incurring prohibitive computational costs.…”