Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study

Khemchandani, Yash; Mehtani, Sarvesh; Patil, Vaidehi; Awasthi, Abhijeet; Talukdar, Partha; Sarawagi, Sunita

doi:10.18653/v1/2021.acl-long.105

Cited by 25 publications

(26 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Amrhein and Sennrich, 2020) studied how transliter-ation improved NMT and came to the conclusion that transliteration offered significant improvement for low-resource languages with different scripts. (Khemchandani et al, 2021) showed on Indo-Aryan languages that language relatedness could be exploited through transliteration along with bilingual lexicon-based pseudo-translation and aligned loss to incorporate low-resource languages into pretrained mBERT. (Muller et al, 2021a) showed that for unseen languages, script barrier hindered transfer between low-resource and high-resource languages for MLLMs and transliteration removed this barrier.…”

Section: Motivation and Backgroundmentioning

confidence: 99%

Does Transliteration Help Multilingual Language Modeling?

Moosa¹,

Akhter²,

Habib³

2022

Preprint

View full text Add to dashboard Cite

As there is a scarcity of large representative corpora for most languages, it is important for Multilingual Language Models (MLLM) to extract the most out of existing corpora. In this regard, script diversity presents a challenge to MLLMs by reducing lexical overlap among closely related languages. Therefore, transliterating closely related languages that use different writing scripts to a common script may improve the downstream task performance of MLLMs. In this paper, we pretrain two AL-BERT models to empirically measure the effect of transliteration on MLLMs. We specifically focus on the Indo-Aryan language family, which has the highest script diversity in the world. Afterward, we evaluate our models on the IndicGLUE benchmark. We perform Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity (CLRS) of the models using centered kernel alignment (CKA) on parallel sentences of eight languages from the FLORES-101 dataset. We find that the hidden representations of the transliteration-based model have higher and more stable CLRS scores. Our code is available at Github 1 Hugging Face Hub 2 , 3

show abstract

Section: Motivation and Backgroundmentioning

confidence: 99%

Does Transliteration Help Multilingual Language Modeling?

Moosa¹,

Akhter²,

Habib³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Input Data In the data creation stage, Conneau et al (2020a) propose over-sampling of LRL documents to improve LRL representation in the vocabulary and pre-training steps. Khemchandani et al (2021) specifically target related languages and propose transliteration of LRL documents to the script of related HRL for greater lexical overlap. We deploy both these tricks in this paper.…”

Section: Related Workmentioning

confidence: 99%

“…(Maronikolakis et al, 2021) targets tok-enization compatibility based purely on vocabulary size and does not focus on choosing the tokens that go in the vocabulary. Pre-Training and Adaptation Several previous works have proposed to include additional alignment loss between parallel or pseudo-parallel (Khemchandani et al, 2021) sentences to co-embed HRLs and LRLs. Another approach is to design language-specific Adapter layers (Pfeiffer et al, 2020a,b;Artetxe et al, 2020;Üstün et al, 2020) that can be easily fine-tuned for each new language.…”

Section: Related Workmentioning

confidence: 99%

“…This under-allocation of model capacity results in lower LRL performance (Wu and Dredze, 2020), as mentioned previously. In response, prior research has explored development of region-specific models (Antoun et al;Khanuja et al, 2021), generating vocabulary specific to language clusters (Chung et al, 2020), and exploring relatedness among languages to build better LMs for LRLs (Khemchandani et al, 2021). However, none of these methods have utilized relatedness among languages for better vocabulary generation during multilingual pre-training.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages

Patil¹,

Talukdar²,

Sarawagi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Pre-trained multilingual language models such as mBERT and XLM-R have demonstrated great potential for zero-shot cross-lingual transfer to low web-resource languages (LRL). However, due to limited model capacity, the large difference in the sizes of available monolingual corpora between high web-resource languages (HRL) and LRLs does not provide enough scope of co-embedding the LRL with the HRL, thereby affecting the downstream task performance of LRLs. In this paper, we argue that relatedness among languages in a language family along the dimension of lexical overlap may be leveraged to overcome some of the corpora limitations of LRLs. We propose Overlap BPE (OBPE), a simple yet effective modification to the BPE vocabulary generation algorithm which enhances overlap across related languages. Through extensive experiments on multiple NLP tasks and datasets, we observe that OBPE generates a vocabulary that increases the representation of LRLs via tokens shared with HRLs. This results in improved zero-shot transfer from related HRLs to LRLs without reducing HRL representation and accuracy. Unlike previous studies that dismissed the importance of token-overlap, we show that in the low-resource related language setting, token overlap matters. Synthetically reducing the overlap to zero can cause as much as a four-fold drop in zero-shot transfer accuracy.

show abstract

“…Previous work has shown the importance of language relatedness for cross-lingual transfer Aharoni et al, 2019;Kudugunta et al, 2019). In the case of Indic languages, orthographic similarity between Indic languages has been utilized to represent data in a common script and improve cross-lingual transfer (Dabre et al, 2018;Goyal et al, 2020b;Khemchandani et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

IndicBART: A Pre-trained Model for Natural Language Generation of Indic Languages

Dabre,

Shrotriya,

Kunchukuttan

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper we present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. Different from existing pre-trained models, In-dicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT for 12 language pairs and extreme summarization for 7 languages using multilingual fine-tuning show that IndicBART is competitive with or better than mBART50 despite containing significantly fewer parameters. Our analyses focus on identifying the impact of script unification (to Devanagari), corpora size as well as multilingualism on the final performance. The IndicBART model is available under the MIT license at https://indicnlp. ai4bharat.org/indic-bart.

show abstract

Exploiting Language Relatedness for Low Web-Resource Language Model Adaptation: An Indic Languages Study

Cited by 25 publications

References 24 publications

Does Transliteration Help Multilingual Language Modeling?

Does Transliteration Help Multilingual Language Modeling?

Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages

IndicBART: A Pre-trained Model for Natural Language Generation of Indic Languages

Contact Info

Product

Resources

About