SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC)

Abstract

In this paper, we introduce the first SemEval task on Multilingual and Cross-Lingual Wordin-Context disambiguation (MCL-WiC). This task allows the largely under-investigated inherent ability of systems to discriminate between word senses within and across languages to be evaluated, dropping the requirement of a fixed sense inventory. Framed as a binary classification, our task is divided into two parts. In the multilingual sub-task, participating systems are required to determine whether two target words, each occurring in a different context within the same language, express the same meaning or not. Instead, in the crosslingual part, systems are asked to perform the task in a cross-lingual scenario, in which the two target words and their corresponding contexts are provided in two different languages. We illustrate our task, as well as the construction of our manually-created dataset including five languages, namely Arabic, Chinese, English, French and Russian, and the results of the participating systems. Datasets and results are available at: https://github.com/ SapienzaNLP/mcl-wic.

ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD

“…The WSD task aims at determining which sense (or gloss) a word in context denotes from a given set of senses. It is also worth noting that these two tasks are not the same as the Word-In-Context (WIC) task Martelli et al, 2021), which aims at determining whether a target word has the same sense in two given contexts.…”

6E.001 Physical injuries and burns among refugees in Lebanon: prevention, programs, and policies

“…Given our Arabic WiC disambigation method described in Section 3, and given the SemEval multilingual dev.ar-ar dataset provided by SemEval-2021 (Martelli et al, 2021), four classification experiments were conducted using the cosine similarity and based on the two Word2Vec models and the two Lemma2Vec models. The objective is to tune the following parameters for each model: context size (ranging from 1 to 10), threshold (we determined empirically the range from 0.55 to 0.85 with 0. the following to find the high F1-scores for T and F: For each context size (between 1 and 10) and for each value of the stop words (yes or no) we plotted 8 line plots (4 for T and 4 for F) for each of the four pooling operations (mean, max, min and std) and for threshold ranging from 0.55 to 0.85 (i.e., 20 plots for each model, resulting 80 plots).…”

“…Experiments presented in this paper are part of the SemEval shared task for Word-in-Context disambiguation (Martelli et al, 2021).…”

Uppsala NLP at SemEval-2021 Task 2: Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguation

“…SemEval-2021 Task 2: Multilingual and Crosslingual Word-in-Context Disambiguation (MCL-WiC) (Martelli et al, 2021) is an extension from WiC (Pilehvar and Camacho-Collados, 2019), a shared task at the IJCAI-19 SemDeep workshop (SemDeep-5). WiC was proposed as a benchmark to evaluate context-sensitive word representations.…”