Meta-XNLG: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation: A Meta-Learning Approach Based on Language Clustering for Zero-Shot Cross-Lingual Transfer and Generation

Abstract

Recently, the NLP community has witnessed a rapid advancement in multilingual and crosslingual transfer research where the supervision is transferred from high-resource languages (HRLs) to low-resource languages (LRLs). However, the cross-lingual transfer is not uniform across languages, particularly in the zeroshot setting. Towards this goal, one promising research direction is to learn shareable structures across multiple tasks with limited annotated data. The downstream multilingual applications may benefit from such a learning setup as most of the languages across the globe are low-resource and share some structures with other languages. In this paper, we propose a novel meta-learning framework (called Meta-X NLG ) to learn shareable structures from typologically diverse languages based on metalearning and language clustering. This is a step towards uniform cross-lingual transfer for unseen languages. We first cluster the languages based on language representations and identify the centroid language of each cluster. Then, a meta-learning algorithm is trained with all centroid languages and evaluated on the other languages in the zero-shot setting. We demonstrate the effectiveness of this modeling on two NLG tasks (Abstractive Text Summarization and Question Generation), 5 popular datasets and 30 typologically diverse languages. Consistent improvements over strong baselines demonstrate the efficacy of the proposed framework. The careful design of the model makes this end-to-end NLG setup less vulnerable to the accidental translation problem, which is a prominent concern in zero-shot cross-lingual NLG tasks. Cluster-1(14)Cluster-2(8)

Proceedings of the Big Picture Workshop

“…Details of all results and observations are included in the MetaX NLG original paper (Maurya and Desarkar, 2022). In summary, based on automated scores, the proposed MetaX NLG model outperformed baselines in 30 out of 33 LRLs for the ATS task and in 18 out of 19 LRLs for the QG task.…”

“…Unlike NLU tasks, the zero-shot NLG is a more challenging setup due to the typological diversities of languages and CF/AT problems. We refer to this framework as MetaX NLG 6 (Maurya and Desarkar, 2022), a framework for effective cross-lingual transfer and gen-eration based on language clustering and Model-Agnostic Meta-Learning (MAML) algorithm (Finn et al, 2017).…”

“…2. We proposed the first meta-learning approach for cross-lingual generation in LRLs (MetaX NLG ; Maurya and Desarkar (2022)). It is based on language clustering to improve the cross-lingual transfer, even for distant LRLs.…”

Clustering Pseudo Language Family in Multilingual Translation Models with Fisher Information Matrix

“…* Co-first and Corresponding Author Current academic investigations in this domain generally fall into two distinct categories: Firstly, the integration of diverse prior knowledge, which incorporates the utilization of various forms of existing knowledge and language family classifications, as evidenced in research such as Bakker et al (2009); Chen and Gerdes (2017). Secondly, the exploration and computation of language similarity center on examining language representations and their comparative resemblances (Tan et al, 2019;Oncevay et al, 2020;Maurya and Desarkar, 2022).…”

Towards Low-resource Language Generation with Limited Supervision

“…2. We proposed the first meta-learning approach for cross-lingual generation in LRLs (MetaX NLG ; Maurya and Desarkar (2022)…”