Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Caswell, Isaac; Kreutzer, Julia; Wang, Lisa; Wahab, Ahsan; Esch, Daan van; Ulzii-Orshikh, Nasanbayar; Tapo, Allahsera Auguste; Subramani, Nishant; Sokolov, Artem; Sikasote, Claytone; Setyawan, Monang; Sarin, Supheakmungkol; Samb, Sokhar; Sagot, Benoît; Rivera, Clara E.; Rios, Annette; Papadimitriou, Isabel; Osei, Salomey; Suárez, Pedro Javier Ortiz; Orife, Iroro; Ogueji, Kelechi; Niyongabo, Rubungo Andre; Nguyen, Toan Q.; Müller, Mathias; Müller, André; Muhammad, Shamsuddeen Hassan; Muhammad, N Sakib; Mnyakeni, Ayanda; Mirzakhalov, Jamshidbek; Matangira, Tapiwanashe; Leong, Colin; Lawson, Nze; Kudugunta, Sneha; Jernite, Yacine; Jenny, Mathias; Fırat, Orhan; Dossou, Bonaventure F. P.; Dlamini, Sakhile; Silva, Nisansa de; Ballı, Sakine Çabuk; Biderman, Stella; Battisti, Alessia; Baruwa, Ahmed; Bapna, Ankur; Baljekar, Pallavi; Azime, Israel Abebe; Awokoya, Ayodele; Ataman, Duygu; Ahia, Orevaoghene; Ahia, Oghenefego; Agrawal, Sweta; Adeyemi, Mofetoluwa

doi:10.1162/tacl_a_00447

Cited by 60 publications

(49 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the authors of mC4 still perform text deduplication, language detection and bad words removal, they lower the language detection threshold to 70% and omit other useful heuristics due to the great variability across the covered character systems, such as filtering sentences having non-standard end-of-sentence punctuation. As a consequence, the resulting corpus has an overall lower quality, with a recent study finding 16% of examples in a random sample of mC4 being associated to the wrong language tag, and 11% of them not containing any linguistic information (Kreutzer et al, 2022).…”

Section: Data and Model Pretrainingmentioning

confidence: 99%

IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

Sarti¹,

Nissim²

2022

Preprint

View full text Add to dashboard Cite

The T5 model and its unified text-to-text paradigm contributed in advancing the stateof-the-art for many natural language processing tasks. While some multilingual variants of the T5 model have recently been introduced, their performances were found to provide suboptimal performances for languages other than English if compared to monolingual variants. We are motivated by these findings to introduce IT5, the first family of encoderdecoder transformer models pretrained specifically on Italian. We perform a thorough cleaning of a web-crawled Italian corpus including more than 40 billion words and use it to pretrain three IT5 models of different sizes. The performance of IT5 models and their multilingual counterparts is then evaluated on a broad range of natural language understanding and generation benchmarks for Italian. We find the monolingual IT5 models to provide the best scale-to-performance ratio across tested models, consistently outperforming their multilingual counterparts and setting a new state-of-the-art for most Italian conditional language generation tasks. Code, data and checkpoints are made available: https: //github.com/gsarti/it5

show abstract

Section: Data and Model Pretrainingmentioning

confidence: 99%

IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

Sarti¹,

Nissim²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…One of the main repositories of parallel data for MT is OPUS (Tiedemann, 2012), which includes many multilingual parallel datasets ranging in domains, languages and sizes. Catalan is included in many of these large webcrawled datasets, however, as Kreutzer et al (2021) point out, most data coming from online sources is of poor quality. Hence, the importance of high quality data curation.…”

Section: Related Workmentioning

confidence: 99%

“…Nonetheless, we are aware that the quality of the datasets varies greatly, since an automatic alignment and manual revision yield very different results. CCaligned, for instance, has been shown to have poor quality (Kreutzer et al, 2021). 2021b) pipeline to process the WARC files obtained from the crawling.…”

Section: Machine Translationmentioning

confidence: 99%

Sequence-to-Sequence Resources for Catalan

Gibert¹,

Kharitonova²,

Figueras³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we introduce sequence-to-sequence language resources for Catalan, a moderately under-resourced language, towards two tasks, namely: Summarization and Machine Translation (MT). We present two new abstractive summarization datasets in the domain of newswire. We also introduce a parallel Catalan↔English corpus, paired with three different brand new test sets. Finally, we evaluate the data presented with competing state of the art models, and we develop baselines for these tasks using a newly created Catalan BART. We release the resulting resources of this work under open license to encourage the development of language technology in Catalan.

show abstract

“…To deal with the demands of deep learning, data curators and researchers have turned to enormous internet-scraped datasets such as Common Crawl Corpus or WebText. As these unstructured corpora become larger, the risk of them containing harmful content increases, and the larger the dataset, the more difficult it is for humans to explore what is in the dataset and audit for quality or toxicity (Hanna and Park, 2020;Luccioni and Viviano, 2021;Kreutzer et al, 2022).…”

Section: Harms and Risks In Nlp Datamentioning

confidence: 99%

“…However, work in seemingly unrelated NLP domains (e.g. NLG, part-of-speech tagging, or semantic search) may still encounter spurious harms in datasets, especially if these are large-scale and scraped from internet sources (Luccioni and Viviano, 2021;Dodge et al, 2021;Kreutzer et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Handling and Presenting Harmful Text

Derczynski¹,

Kirk²,

Birhane³

et al. 2022

Preprint

View full text Add to dashboard Cite

Textual data can pose a risk of serious harm. These harms can be categorised along three axes: (1) the harm type (e.g. misinformation, hate speech or racial stereotypes) (2) whether it is elicited as a feature of the research design from directly studying harmful content (e.g. training a hate speech classifier or auditing unfiltered large-scale datasets) versus spuriously invoked from working on unrelated problems (e.g. language generation or part of speech tagging) but with datasets that nonetheless contain harmful content, and (3) who it affects, from the humans (mis)represented in the data to those handling or labelling the data to readers and reviewers of publications produced from the data. It is an unsolved problem in NLP as to how textual harms should be handled, presented, and discussed; but, stopping work on content which poses a risk of harm is untenable. Accordingly, we provide practical advice and introduce HARMCHECK, a resource for reflecting on research into textual harms. We hope our work encourages ethical, responsible, and respectful research in the NLP community.

show abstract

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Cited by 60 publications

References 39 publications

IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

IT5: Large-scale Text-to-text Pretraining for Italian Language Understanding and Generation

Sequence-to-Sequence Resources for Catalan

Handling and Presenting Harmful Text

Contact Info

Product

Resources

About