Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Abstract

We present Samanantar, the largest publicly available parallel corpora collection for Indic languages. The collection contains a total of 49.7 million sentence pairs between English and 11 Indic languages (from two language families). Specifically, we compile 12.4 million sentence pairs from existing, publicly available parallel corpora, and additionally mine 37.4 million sentence pairs from the Web, resulting in a 4× increase. We mine the parallel sentences from the Web by combining many corpora, tools, and methods: (a) Web-crawled monolingual corpora, (b) document OCR for extracting sentences from scanned documents, (c) multilingual representation models for aligning sentences, and (d) approximate nearest neighbor search for searching in a large collection of sentences. Human evaluation of samples from the newly mined corpora validate the high quality of the parallel sentences across 11 languages. Further, we extract 83.4 million sentence pairs between all 55 Indic language pairs from the English-centric parallel corpus using English as the pivot language. We trained multilingual NMT models spanning all these languages on Samanantar which outperform existing models and baselines on publicly available benchmarks, such as FLORES, establishing the utility of Samanantar. Our data and models are available publicly at Samanantar and we hope they will help advance research in NMT and multilingual NLP for Indic languages.

Overview of the Shared Task on Machine Translation in Dravidian Languages

“…In the set I, parallel corpora were collected from Samanantar 8 -a collection of the largest parallel corpora available for Indic languages (Ramesh et al, 2022) and statistics of set I is shown in Table 1. It may be noted that only a small portion is used in this task instead of using whole dataset.…”

Bilingual Tabular Inference: A Case Study on Indic Languages

“…Thus, fact verification in a bilingual setting wherein, the premise is in English and the claim/hypothesis is in an Indic language, is of great significance. Moreover, recent advances in multilingual language models (Khanuja et al, 2021a;Kunchukuttan, 2020), datasets (Roark et al, 2020;Ramesh et al, 2022), and translation systems (Ramesh et al, 2022) for Indian languages have enabled quality examination of several Indic NLU tasks which serves as additional motivation to evaluate the task of bTNLI for Indic languages before other low resource languages.…”

“…To construct EI-INFOTABS, we machine translated the English hypotheses provided in INFOTABS to 11 major Indian languages as described earlier. We use IndicTrans (Ramesh et al, 2022), an open-sourced state-of-the-art Indic NMT model. IndicTrans is trained on the Samanantar dataset (Ramesh et al, 2022), which is the largest publicly available parallel corpus for Indic languages.…”

“…We use IndicTrans (Ramesh et al, 2022), an open-sourced state-of-the-art Indic NMT model. IndicTrans is trained on the Samanantar dataset (Ramesh et al, 2022), which is the largest publicly available parallel corpus for Indic languages. Moreover, it outperforms (a) commercial NMT systems like Google-Translate 5 and Bing Microsoft Translator 6 , and (b) open-source multilingual models like OPUS-MT (Tiedemann and Thottingal, 2020), mBART50 and mT5 (Xue et al, 2021).…”

Attentive fine-tuning of Transformers for Translation of low-resourced languages @LoResMT 2021

“…Training on fairseq enables competent batching, mixed-precision training, multi-GPU and multi-machine training. IndicTrans (Ramesh et al, 2021), a Transformer-4x multilingual NMT model by AI4Bharat, is trained on the Samanantar dataset. The architecture of our approach is displayed in Fig.…”