Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study

Abstract

Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. Since mBERT is not pre-trained with explicit cross-lingual supervision, transfer performance can further be improved by aligning mBERT with cross-lingual signal. Prior work proposes several approaches to align contextualised embeddings. In this paper we analyse how different forms of cross-lingual supervision and various alignment methods influence the transfer capability of mBERT in zero-shot setting. Specifically, we compare parallel corpora vs. dictionary-based supervision and rotational vs. fine-tuning based alignment methods. We evaluate the performance of different alignment methodologies across eight languages on two tasks: Name Entity Recognition and Semantic Slot Filling. In addition, we propose a novel normalisation method which consistently improves the performance of rotation-based alignment including a notable 3% F1 improvement for distant and typologically dissimilar languages. Importantly we identify the biases of the alignment methods to the type of task and proximity to the transfer language. We also find that supervision from parallel corpus is generally superior to dictionary alignments.

Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

“…Contextual representations can be obtained through multilingual pre-training, which encodes whole sentence and outputs contextual representation for each word (Devlin et al, 2019;Lample and Conneau, 2019). Due to the rich context information contained in the contextual representations, there are endeavors to align them in different languages (Schuster et al, 2019;Aldarmaki and Diab, 2019;Wang et al, 2020;Kulshreshtha et al, 2020;Cao et al, 2020).…”

Large Language Models and Low-Resource Languages: An Examination of Armenian NLP

“…Cross-lingual Transfer: In an endeavor to enhance the cross-lingual transfer abilities of Multilingual BERT (mBERT), Kulshreshtha et al (2020) managed to improve its performance. The researchers achieved this by aligning mBERT with cross-lingual signals, employing parallel corpora supervision, and fine-tuning the alignment.…”

“…In a related effort, Ter-Hovhannisyan and Avetisyan (2022) utilized the transformer-based XLM-RoBERTa model for cross-lingual sentence alignment, highlighting its effectiveness in multilingual contexts. Additionally, Kulshreshtha et al (2020) augmented the cross-lingual transfer abilities of Multilingual BERT (mBERT) to achieve superior performance in language transfer tasks.…”

Translation Performance from the User’s Perspective of Large Language Models and Neural Machine Translation Systems

“…In addition, BERT has been incorporated into NMT models as a pre-training mechanism, leading to improved translation quality in various settings [24]. Moreover, several BERT-based models, such as mBERT (multilingual BERT) and XLM-R (Cross-lingual Language Model-RoBERTa), have been developed to handle multilingual and cross-lingual tasks [25,26].…”