XLPT-AMR: Cross-Lingual Pre-Training via Multi-Task Learning for Zero-Shot AMR Parsing and Text Generation

Abstract

Due to the scarcity of annotated data, Abstract Meaning Representation (AMR) research is relatively limited and challenging for languages other than English. Upon the availability of English AMR dataset and English-to-X parallel datasets, in this paper we propose a novel cross-lingual pre-training approach via multi-task learning (MTL) for both zeroshot AMR parsing and AMR-to-text generation. Specifically, we consider three types of relevant tasks, including AMR parsing, AMR-to-text generation, and machine translation. We hope that knowledge gained while learning for English AMR parsing and text generation can be transferred to the counterparts of other languages. With properly pretrained models, we explore four different finetuning methods, i.e., vanilla fine-tuning with a single task, one-for-all MTL fine-tuning, targeted MTL fine-tuning, and teacher-studentbased MTL fine-tuning. Experimental results on AMR parsing and text generation of multiple non-English languages demonstrate that our approach significantly outperforms a strong baseline of pre-training approach, and greatly advances the state of the art. In detail, on LDC2020T07 we have achieved 70.45%, 71.76%, and 70.80% in Smatch F1 for AMR parsing of German, Spanish, and Italian, respectively, while for AMR-to-text generation of the languages, we have obtained 25.69, 31.36, and 28.42 in BLEU respectively. We make our code available on github https:// github.com/xdqkid/XLPT-AMR.

Spanish Abstract Meaning Representation: Annotation of a General Corpus

“…After fine-tuning t5wtense, we see a marked improvement in performance, increasing in BERTscore by approximately 8.8% absolute (11.86% relative improvement). Current state-of-the-art cross-lingual generation (Xu et al, 2021) achieves a BERTscore of 0.8534 on the same test set, 7 which indicates that by fine-tuning on only 376 Spanish AMR annotations, we are able to achieve results close to the current best performing model. 8 The marked improvement resulting from our fine-tuning demonstrates the utility of our corpus and suggests incorporating our data into more sophisticated generation or parsing models can lead to greater improvements.…”

“…An emphasis on the man's ownership of the letter elicits the :poss role, whereas emphasizing the letter's creation by the man elicits the :source role. t5wtense 0.7389 Fine-tuned t5wtense 0.8265 XLPT-AMR 0.8534 Table 3: BERTscore results for: the output of the t5wtense generation model without any fine-tuning, t5wtense after fine-tuning with our data, and the state-of-the-art XLPT-AMR cross-lingual AMR generation model (Xu et al, 2021) on our test split.…”

A Knowledge-Enhanced Adversarial Model for Cross-lingual Structured Sentiment Analysis

“…Recently, multi-lingual pre-trained models such as mBERT [9] and XLM [10] have increasingly attracted attention in various polyglot tasks [11], [12]. Mulcaire et al [13] adopted multi-lingual contextualized word embedding for structure extraction tasks, such as semantic role labeling, dependency parsing, and named-entity recognition.…”

Maximum Bayes Smatch Ensemble Distillation for AMR Parsing

“…Cai et al (2021b) propose to use bilingual input to enable a model to predict more accurate AMR concepts. Xu et al (2021) propose a crosslingual pretraining approach via multitask learning for AMR parsing. Cai et al (2021a) propose to use noisy knowledge distillation for multilingual AMR parsing.…”