Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

Abstract

The emergent cross-lingual transfer seen in multilingual pretrained models has sparked significant interest in studying their behavior. However, because these analyses have focused on fully trained multilingual models, little is known about the dynamics of the multilingual pretraining process. We investigate when these models acquire their in-language and crosslingual abilities by probing checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. In contrast, when the model learns to transfer cross-lingually depends on the language pair. Interestingly, we also observe that, across many languages and tasks, the final, converged model checkpoint exhibits significant performance degradation and that no one checkpoint performs best on all languages. Taken together with our other findings, these insights highlight the complexity and interconnectedness of multilingual pretraining.

How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning

“…Thus we conclude that on average the models gradually start to perform more cross-lingual sharing as fine-tuning progresses. Moreover, in line with previous findings (Blevins et al, 2022), we observe that the amount of cross-lingual sharing between different language-pairs fluctuates during fine-tuning (see Appendix C for results). To test whether the ranked influence scores between epochs are statistically significantly different, we apply the Wilcoxon signed-rank test (Wilcoxon, 1992), and confirm that between all epochs this holds true (p-value < 0.05).…”

“…While in previous results we reported the sum of these scores, we now analyze them separately per fine-tuning epoch. Blevins et al (2022) study cross-lingual pretraining dynamics of multilingual models to see when crosslingual sharing emerges. We instead study whether different patterns emerge when looking at language influence across fine-tuning.…”