XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

Ponti, Edoardo Maria; Glavaš, Goran; Majewska, Olga; Liu, Qianchu; Vulić, Ivan; Korhonen, Anna

doi:10.18653/v1/2020.emnlp-main.185

Cited by 72 publications

(53 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another multilingual data set, PAWS-X (Yang et al 2019), focused on the paraphrase identification task and was created translating the original English PAWS (Zhang, Baldridge, and He 2019) into 6 languages. XCOPA (Ponti et al 2020) is a crosslingual data set for the evaluation of crosslingual causal commonsense reasoning, obtained through translation of the English COPA data (Roemmele, Bejan, and Gordon 2011) to 11 target languages. A large number of tasks have been recently integrated into unified multilingual evaluation suites: XTREME (Hu et al 2020) and XGLUE (Liang et al 2020).…”

Section: Previous Work and Evaluation Datamentioning

confidence: 99%

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

Vulić

Baker

Ponti

et al. 2021

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering data sets for 12 typologically diverse languages, including major languages (e.g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e.g., Welsh, Kiswahili). Each language data set is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs, providing a representative coverage of word classes (nouns, verbs, adjectives, adverbs), frequency ranks, similarity intervals, lexical fields, and concreteness levels. Additionally, owing to the alignment of concepts across languages, we provide a suite of 66 cross-lingual semantic similarity data sets. Due to its extensive size and language coverage, Multi-SimLex provides entirely novel opportunities for experimental evaluation and analysis. On its monolingual and cross-lingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and cross-lingual representation models, including static and contextualized word embeddings (such as fastText, monolingual and multilingual BERT, XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised cross-lingual word embeddings. We also present a step-by-step data set creation protocol for creating consistent, Multi-Simlex -style resources for additional languages. We make these contributions - the public release of Multi-SimLex data sets, their creation protocol, strong baseline results, and in-depth analyses which can be be helpful in guiding future developments in multilingual lexical semantics and representation learning - available via a website which will encourage community effort in further expansion of Multi-Simlex to many more languages. Such a large-scale semantic resource could inspire significant further advances in NLP across languages.

show abstract

Section: Previous Work and Evaluation Datamentioning

confidence: 99%

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

Vulić

Baker

Ponti

et al. 2021

Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a result, many recent benchmarking and dataset creation efforts in NLU develop and focus on tasks that are inherently multilingual or which explore cross-lingual transfer. For example, XTREME (Hu et al, 2020) introduces a benchmark covering 40 languages across multiple NLU and retrieval tasks, XCOPA (Ponti et al, 2020) is a commonsense reasoning dataset for eleven languages, and MLQA ) is a dataset for extractive question answering across seven languages. We can observe a similar recent trend in natural language generation, where ML-Sum and WikiLingua were created as multilingual summarization datasets.…”

Section: Increasing Multilingualism Of Nlg Researchmentioning

confidence: 99%

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

Gehrmann¹,

Adewumi²,

Aggarwal³

et al. 2021

Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)

View full text Add to dashboard Cite

We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with wellestablished, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.

show abstract

“…Commonsense datasets in multiple languages or languages other than English have also been created recently. XCOPA (Ponti et al, 2020) In the aspect of commonsense reasoning, our dataset is different from the mentioned commonsense datasets in that we detect and annotate errors in machine-generated texts, which violates common sense, rather than creating examples to examine the commonsense reasoning ability of machines.…”

Section: Commonsense Datasetsmentioning

confidence: 99%