Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Ramesh, Gowtham; Doddapaneni, Sumanth; Bheemaraj, Aravinth; Jobanputra, Mayank; AK, Raghavan; Sharma, Ajitesh; Sahoo, Sujit Kumar; Diddee, Harshita; Mahalakshmi, J; Kakwani, Divyanshu; Kumar, Nirbhay; Pradeep, Aswin; Nagaraj, Srihari; Kumar, Deepak; Raghavan, V.; Kunchukuttan, Anoop; Kumar, Pratyush; Khapra, Mitesh Shantadevi

doi:10.48550/arxiv.2104.05596

Cited by 8 publications

(7 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MT systems are generally modeled as monolingual or multilingual models based on the (i) availability of large training corpus (ii) similarities between languages. [5,73,84].…”

Section: Machine Translationmentioning

confidence: 99%

A survey in Adversarial Defences and Robustness in NLP

Goyal¹,

Doddapaneni²,

Khapra³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, it has been seen that deep neural networks are lacking robustness and are likely to break in case of adversarial perturbations in input data. Strong adversarial attacks are proposed by various authors for computer vision and Natural Language Processing (NLP). As a counter-effort, several defense mechanisms are also proposed to save these networks from failing. In contrast with image data, generating adversarial attacks and defending these models is not easy in NLP because of the discrete nature of the text data. However, numerous methods for adversarial defense are proposed of late, for different NLP tasks such as text classification, named entity recognition, natural language inferencing, etc. These methods are not just used for defending neural networks from adversarial attacks, but also used as a regularization mechanism during training, saving the model from overfitting. The proposed survey is an attempt to review different methods proposed for adversarial defenses in NLP in the recent past by proposing a novel taxonomy. This survey also highlights the fragility of the advanced deep neural networks in NLP and the challenges in defending them.

show abstract

“…MT systems are generally modeled as monolingual or multilingual models based on the (i) availability of large training corpus (ii) similarities between languages. [5,73,84].…”

Section: Machine Translationmentioning

confidence: 99%

A survey in Adversarial Defences and Robustness in NLP

Goyal¹,

Doddapaneni²,

Khapra³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Datasets We will consider a total of 51 Englishcentric language pairs. 6 We gather this data from WMT, the OPUS 100 (Zhang et al, 2020) dataset, Paracrawl 7 and Samanantar (Ramesh et al, 2021).…”

Section: Model Configurationsmentioning

confidence: 99%

Using natural language prompts for machine translation

García¹,

Fırat²

2022

Preprint

View full text Add to dashboard Cite

We explore the use of natural language prompts for controlling various aspects of the outputs generated by machine translation models. We demonstrate that natural language prompts allow us to influence properties like formality or specific dialect of the output. We show that using language names to control the output language of multilingual translation models enables positive transfer for unseen language pairs. This unlocks the ability to translate into languages not seen during finetuning by using their English names. We investigate how scale, number of pre-training steps, number of languages in fine-tuning, and language similarity affect this phenomenon.

show abstract

“…Weakly Labeled Data Generation: We propose an annotation-projection-based approach to create weakly labeled data from a large bi-lingual parallel corpus (Ramesh et al 2021). Let {(S i , T i )} N i=1 having N sentences be a parallel corpus of English and a low-resource language, where (S i , T i ) represents a pair of sentences which are translations of each other.…”

Section: Proposed Approachmentioning

confidence: 99%

CL-NERIL: A Cross-Lingual Model for NER in Indian Languages (Student Abstract)

Prabhakar

Majumder

Anand

2022

AAAI

View full text Add to dashboard Cite

Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a target Indian language. We employ a variant of the Teacher-Student model and optimize it jointly on the pseudo labels of the Teacher model and predictions on the generated weakly labeled data. We also present manually annotated test sets for three Indian languages: Hindi, Bengali, and Gujarati. We evaluate the performance of the proposed framework on the test sets of the three Indian languages. Empirical results show a minimum 10% performance improvement compared to the zero-shot transfer learning model on all languages. This indicates that weakly labeled data generated using the proposed annotation projection method in target Indian languages can complement well-annotated source language data to enhance performance. Our code is publicly available at https://github.com/aksh555/CL-NERIL.

show abstract

Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages

Cited by 8 publications

References 26 publications

A survey in Adversarial Defences and Robustness in NLP

A survey in Adversarial Defences and Robustness in NLP

Using natural language prompts for machine translation

CL-NERIL: A Cross-Lingual Model for NER in Indian Languages (Student Abstract)

Contact Info

Product

Resources

About