Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings

Abstract

Recent studies have proposed different methods to improve multilingual word representations in contextualized settings including techniques that align between source and target embedding spaces. For contextualized embeddings, alignment becomes more complex as we additionally take context into consideration. In this work, we propose using Optimal Transport (OT) as an alignment objective during fine-tuning to further improve multilingual contextualized representations for downstream cross-lingual transfer. This approach does not require word-alignment pairs prior to fine-tuning that may lead to sub-optimal matching and instead learns the word alignments within context in an unsupervised manner. It also allows different types of mappings due to soft matching between source and target sentences. We benchmark our proposed method on two tasks (XNLI and XQuAD) and achieve improvements over baselines as well as competitive results compared to similar recent works.

Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation

“…For tasks involving cross-lingual settings, Nguyen and Luu [29] employed OT distance as a part of the loss function in a knowledge distillation framework for improving the cross-lingual summarization. Alqahtani et al [1] incorporated OT as an alignment objective to improve the multilingual word representations. In this work, we explore transferring the retrieval knowledge in a cross-lingual setting via OT.…”

“…Therefore, we approximate the calculation of 𝐷 ( q, 𝑞) as an optimal transport problem. First, we assign equal mass to the tokens in q and 𝑞 by defining a uniform source probability distribution, 𝜇 𝑠 , on q and a uniform target probability distribution, 𝜇 𝑡 , on 𝑞: 𝜇 𝑠 (𝑖) = 1 𝐿 and 𝜇 𝑡 ( 𝑗) = 1 𝐿 where 1 ≤ 𝑖, 𝑗 ≤ 𝐿. The set of transportation plans between these two distributions is then the set of doubly stochastic matrices P defined as…”