WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia

Abstract

We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 96 languages, including several dialects or low-resource languages. We systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus is freely available. 1 To get an indication on the quality of the extracted bitexts, we train neural MT baseline systems on the mined data only for 1886 languages pairs, and evaluate them on the TED corpus, achieving strong BLEU scores for many language pairs. The WikiMatrix bitexts seem to be particularly interesting to train MT systems between distant languages without the need to pivot through English.

GeBioToolkit: Automatic Extraction of Gender-Balanced Multilingual Corpus of Wikipedia Biographies

“…The most recent initiative is the so-called LASER (Artetxe and Schwenk, 2019b), which relies on vector representations of sentences to extract similar pairs. This toolkit has been used to extract the WikiMatrix corpus (Schwenk et al, 2019) which contains 135 million parallel sentences for 1,620 different language pairs in 85 different languages.…”

“…The corpus alignment module makes use of the text and dictionaries retrieved in the previous step and the LASER toolkit 7 . LASER (Language-Agnostic SEntence Representations) allows to obtain sentence embeddings through a multilingual sentence encoder (Schwenk et al, 2019). Translations can be found then as close pairs (tuples) in the multilingual semantic space.…”

“…All three are highlighted in Figure 2. Even if the Catalan Wikipedia is not as big as the English and Spanish ones, there is a noticeable amount of comparable articles between Spanish and Catalan which translates into significant number of parallel sentences - Schwenk et al (2019) extracted in WikiMatrix 3,377 k sentences for en-es, 1,580 k sentences for ca-es and 210 k sentences for en-ca from the full editions. GeBioToolkit extracts multilingually aligned parallel sentences, so it is interesting to study also the union of languages.…”

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

“…Curating such datasets relies on the Web sites giving clues about the language of their contents (e.g., a language identifier in the URL) and on automatic language classification (LangID). It is commonly known that these automatically crawled and filtered datasets tend to have overall lower quality than hand-curated collections , but their quality is rarely measured directly, and is rather judged through the improvements they bring to downstream applications (Schwenk et al, 2021).…”

“…To shed light on the quality of data crawls for the lowest resource languages, we perform a manual data audit for 230 per-language subsets of five major crawled multilingual datasets: 1 CCAligned (El- Kishky et al, 2020), ParaCrawl (Esplà et al, 2019;Bañón et al, 2020), Wiki-Matrix (Schwenk et al, 2021), OSCAR (Ortiz Suárez et al, 2019;, and mC4 (Xue et al, 2021). We propose solutions for effective, low-effort data auditing (Section 4), including an error taxonomy.…”