Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources

Yu, Xinyan; Chatterjee, Trina; Asai, Akari; Hu, Junjie; Choi, Eunsol

doi:10.18653/v1/2022.findings-emnlp.273

Cited by 6 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite the plethora of works on multilingual prompting, little to no African languages are usually contained in the evaluation sets of nearly all of these works. When present, they are often obtained by translating the existing datasets of other languages (Yu et al, 2022) This method has been shown to contain artifacts that can inflate the performance of models evaluated on such datasets (Artetxe et al, 2020). Ahuja et al ( 2023) performs a comprehensive evaluation of the GPT models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages.…”

Section: Related Workmentioning

confidence: 99%

Ìgbélárugẹ Èdè: Akinwumi Isola’s Model for Promoting African Languages

Ojo¹

2021

YSR

View full text Add to dashboard Cite

The ever-popular discussion in African literary circles is critically about language choices that African writers make in their creative endeavors. This is part of this write-up’s focus plus the plight of African languages with attention to the benefit and challenges for their empowerment. We set out to achieve two goals in this essay; first contributing to the ongoing discussions on African mother tongues, their vital roles in African literatures while characterizing pointers on proficiency and performance. Second, considering the use of Yoruba language in creative works of late Akínwùmí Oròjídé Iṣọ̀lá. Expectedly, the latter goal will exemplify the importance of indigenous languages to African writers. In pursuance of these dual goals, it is critical to highlight areas in which African writers, especially those writing in their native African languages, have endured to play crucial roles in promotion of African languages. These highlighted areas go beyond now fashionable and expressed goal of focusing on literature in African languages (splendor in African languages) onto push for fairness for languages and their speakers (linguistic justice).

show abstract

Section: Related Workmentioning

confidence: 99%

Ìgbélárugẹ Èdè: Akinwumi Isola’s Model for Promoting African Languages

Ojo¹

2021

YSR

View full text Add to dashboard Cite

show abstract

“…The study of bilingualism has long been a topic of interest among linguists (Yu et al, 2022;Hoffmann, 2014), as it provides insight into the mechanisms of language acquisition and processing. Furthermore, research on multilingualism has contributed to the development of more effective machine learning models, such as neural translation systems (Zou et al, 2013).…”

Section: Introductionmentioning

confidence: 99%

Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs

Zhang,

Li,

Hauer

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Large language models (LLMs) have demonstrated exceptional natural language understanding abilities, and have excelled in a variety of natural language processing (NLP) tasks. Despite the fact that most LLMs are trained predominantly on English, multiple studies have demonstrated their capabilities in a variety of languages. However, fundamental questions persist regarding how LLMs acquire their multilingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing how they use LLMs and interpret their output. In this work, we propose a systematic way of qualitatively and quantitatively evaluating the multilingual capabilities of LLMs. We investigate the phenomenon of cross-language generalization in LLMs, wherein limited multilingual training data leads to advanced multilingual capabilities.To accomplish this, we employ a novel prompt back-translation method. The results demonstrate that LLMs, such as GPT, can effectively transfer learned knowledge across different languages, yielding relatively consistent results in translation-equivariant tasks, in which the correct output does not depend on the language of the input. However, LLMs struggle to provide accurate results in translation-variant tasks, which lack this property, requiring careful user judgment to evaluate the answers.

show abstract

“…Advances in multilingual natural language processing (NLP) technologies (Dabre et al, 2020;Hedderich et al, 2021) have raised the enticing possibilities of NLP systems that benefit all people around the world. However, at the same time, studies into the state of multilingual NLP have demonstrated stark differences in the amount of resources available (Joshi et al, 2020;Yu et al, 2022) and performance of existing NLP systems (Blasi et al, 2022;Khanuja et al, 2023;Ahia et al, 2023).…”

Section: Introductionmentioning

confidence: 99%

GlobalBench: A Benchmark for Global Progress in Natural Language Processing

Song,

Khanuja,

Liu

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist. Arguably, these are due to uneven resource allocation and sub-optimal incentives to work on less resourced languages. To track and further incentivize the global development of equitable language technology, we introduce GlobalBench. Prior multilingual benchmarks are static and have focused on a limited number of tasks and languages. In contrast, Glob-alBench is an ever-expanding collection that aims to dynamically track progress on all NLP datasets in all languages. Rather than solely measuring accuracy, GlobalBench also tracks the estimated perspeaker utility and equity of technology across all languages, providing a multi-faceted view of how language technology is serving people of the world. Furthermore, GlobalBench is designed to identify the most under-served languages, and rewards research efforts directed towards those languages. At present, the most under-served languages are the ones with a relatively high population, but nonetheless overlooked by composite multilingual benchmarks (like Punjabi, Portuguese, and Wu Chinese). Currently, Global-Bench covers 966 datasets in 190 languages and includes 1,128 system submissions spanning 62 languages. 1

show abstract

Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources

Cited by 6 publications

References 0 publications

Ìgbélárugẹ Èdè: Akinwumi Isola’s Model for Promoting African Languages

Ìgbélárugẹ Èdè: Akinwumi Isola’s Model for Promoting African Languages

Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs

GlobalBench: A Benchmark for Global Progress in Natural Language Processing

Contact Info

Product

Resources

About