Multilingual BERT Post-Pretraining Alignment

Pan, Lin; Hang, Chung-Wei; Qi, Haode; Shah, Abhishek; Potdar, Saloni; Yu, Mo

doi:10.18653/v1/2021.naacl-main.20

Cited by 20 publications

(17 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although we can have different loss functions to optimise XLM-K, we choose contrastive learning due to its promising results in both visual representations (He et al 2020;Chen et al 2020) and cross-lingual pre-training (Chi et al 2021;Pan et al 2021). Intuitively, by distinguishing the positive sample from the negative samples using the contrastive loss, the model stores expressive knowledge acquired from the structure data.…”

Section: Joint Pre-training Objectivementioning

confidence: 99%

See 1 more Smart Citation

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Jiang

Liang

Chen

et al. 2022

AAAI

View full text Add to dashboard Cite

Cross-lingual pre-training has achieved great successes using monolingual and bilingual plain text corpora. However, most pre-trained models neglect multilingual knowledge, which is language agnostic but comprises abundant cross-lingual structure alignment. In this paper, we propose XLM-K, a cross-lingual language model incorporating multilingual knowledge in pre-training. XLM-K augments existing multilingual pre-training with two knowledge tasks, namely Masked Entity Prediction Task and Object Entailment Task. We evaluate XLM-K on MLQA, NER and XNLI. Experimental results clearly demonstrate significant improvements over existing multilingual language models. The results on MLQA and NER exhibit the superiority of XLM-K in knowledge related tasks. The success in XNLI shows a better cross-lingual transferability obtained in XLM-K. What is more, we provide a detailed probing analysis to confirm the desired knowledge captured in our pre-training regimen. The code is available at https://github.com/microsoft/Unicoder/tree/master/pretraining/xlmk.

show abstract

Section: Joint Pre-training Objectivementioning

confidence: 99%

“…The results on MLQA are shown in Table 1, we compare our model with mBERT (Lewis et al 2020), XLM (Lewis et al 2020), mBERT + PPA (Pan et al 2021), Unicoder (Huang et al 2019) and XLM-R base (Conneau et al 2020). Since F1 and EM scores have similar observations, we take F1 scores for analysis:…”

Section: Downstream Task Evaluationmentioning

confidence: 99%

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Jiang

Liang

Chen

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…InfoXLM (Chi et al 2020b) proposed a pre-training task based on contrastive learning, from an information-theoretic perspective. Pan et al (2020) also introduced an alignment method based on contrastive learning. Cao, Kitaev, and Klein (2020) proposed an explicit word-level alignment procedure.…”

Section: Related Workmentioning

confidence: 99%

Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training

Yang¹,

Cui²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multilingual pre-trained language models (MPLMs) not only can handle tasks in different languages but also exhibit surprising zero-shot cross-lingual transferability. However, MPLMs usually are not able to achieve comparable supervised performance on rich-resource languages compared to the state-of-the-art monolingual pre-trained models. In this paper, we aim to improve the multilingual model's supervised and zero-shot performance simultaneously only with the resources from supervised languages. Our approach is based on transferring knowledge from high-performance monolingual models with a teacher-student framework. We let the multilingual model learn from multiple monolingual models simultaneously. To exploit the model's cross-lingual transferability, we propose MBLM (multi-branch multilingual language model), a model built on the MPLMs with multiple language branches. Each branch is a stack of transformers. MBLM is trained with the zero-shot-aware training strategy that encourages the model to learn from the mixture of zero-shot representations from all the branches. The results on two crosslingual classification tasks show that, with only the task's supervised data used, our method improves both the supervised and zero-shot performance of MPLMs.

show abstract

“…Retrieval P@1 With the word-level contrastive objective, we observed significant BLEU score improvements on language pairs such as en-ro, en-et and en-my for mBART FT as presented in Table 2. However, noisy word pairs (Pan et al, 2021a) extracted via word alignment toolkits leads to poor supervision signals for improving sentence retrieval P@1, which in turn prevents some language pairs such as en-kk from exhibiting BLEU improvements. We found that for en-kk, the numbers of extracted word pairs per sentence by word2word and FastAlign are 1.0 and 2.2, respectively.…”

Section: Word-level Contrastive Objective and Sentencementioning

confidence: 99%

When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?

Mao¹,

Chu²,

Dabre³

et al. 2022

Preprint

View full text Add to dashboard Cite

Word alignment has proven to benefit many-tomany neural machine translation (NMT). However, high-quality ground-truth bilingual dictionaries were used for pre-editing in previous methods, which are unavailable for most language pairs. Meanwhile, the contrastive objective can implicitly utilize automatically learned word alignment, which has not been explored in many-to-many NMT. This work proposes a word-level contrastive objective to leverage word alignments for many-to-many NMT. Empirical results show that this leads to 0.8 BLEU gains for several language pairs. Analyses reveal that in many-to-many NMT, the encoder's sentence retrieval performance highly correlates with the translation quality, which explains when the proposed method impacts translation. This motivates future exploration for many-to-many NMT to improve the encoder's sentence retrieval performance.2 Word-level Contrastive Learning for Many-to-many NMT Inspired by the contrastive learning framework and the sentence-level con-

show abstract

Multilingual BERT Post-Pretraining Alignment

Cited by 20 publications

References 19 publications

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge

Cross-Lingual Text Classification with Multilingual Distillation and Zero-Shot-Aware Training

When do Contrastive Word Alignments Improve Many-to-many Neural Machine Translation?

Contact Info

Product

Resources

About