ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation

Abstract

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple lowresource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised crosslingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results. We experimented with few-shot training (with 1000 supervised data-points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analysis to demonstrate the robustness of ZmBART.

Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks

“…• Freezing of decoder and embeddings (Maurya et al, 2021): only weights in the encoder are finetuned. The motivation behind this approach is that the decoder should keep capabilities of generating in various languages while the encoder will adapt the model to the task;…”

“…• Mixing-in self-supervised data for target languages (Lester et al, 2021;Maurya et al, 2021): during finetuning, task data instances in source language will be alternated with self-supervised data instances in target language. The motivation is that such a mixing will preserve model's capability of generation in target languages;…”

“…While the described setting was broadly studied for natural language understanding tasks (Xue et al, 2021;Conneau et al, 2020;Artetxe et al, 2020a;Pires et al, 2019;Wu and Dredze, 2019;Pfeiffer et al, 2020), work on zero-shot cross-lingual gen-eration is more limited (Vu et al, 2022;Pfeiffer et al, 2023;Maurya et al, 2021;Li and Murray, 2023). Previous work highlight two main problems arising in this scenario: producing incoherent or irrelevant answers, and generating text in a wrong language.…”

Domain structure-based transfer learning for cross-domain word representation

“…As is discussed in prior sections, zero-shot cross-lingual generation with mPLM [19,27] suffers much from catastrophic forgetting [41,45]. Prior work assumes it is because mPLM has never been exposed to sentences of any other languages especially the target language during its fine-tuning on supervised data of source language [45].…”

“…While mPLM has shown promise in enabling cross-lingual transfer for generation tasks, zero-shot cross-lingual transfer (ZS-XLT) with mPLM suffers much from catastrophic forgetting, where mPLM that has been fine-tuned on the source language is unable to generate fluent sentences in the target language when being evaluated on it [19,27,41,45]. In this work, we would like to investigate if catastrophic forgetting occurs in two other cross-lingual scenarios namely FS-XLT and MTL in the context of dialogue generation given the availability of just a few data examples in target language.…”