Improving AMR Parsing with Sequence-to-Sequence Pre-training

Abstract

In the literature, the research on abstract meaning representation (AMR) parsing is much restricted by the size of human-curated dataset which is critical to build an AMR parser with good performance. To alleviate such data size restriction, pre-trained models have been drawing more and more attention in AMR parsing. However, previous pre-trained models, like BERT, are implemented for general purpose which may not work as expected for the specific task of AMR parsing. In this paper, we focus on sequence-to-sequence (seq2seq) AMR parsing and propose a seq2seq pre-training approach to build pre-trained models in both single and joint way on three relevant tasks, i.e., machine translation, syntactic parsing, and AMR parsing itself. Moreover, we extend the vanilla fine-tuning method to a multi-task learning fine-tuning method that optimizes for the performance of AMR parsing while endeavors to preserve the response of pre-trained models. Extensive experimental results on two English benchmark datasets show that both the single and joint pre-trained models significantly improve the performance (e.g., from 71.5 to 80.2 on AMR 2.0), which reaches the state of the art. The result is very encouraging since we achieve this with seq2seq models rather than complex models.

SGL: Speaking the Graph Languages of Semantic Parsing via Multilingual Translation

“…Continuing this latter direction and arguably closest to our work, Xu et al (2020a) and Bevilacqua et al (2021) show that these models, once paired with adequate pre-training, can behave on par or better than dedicated and more sophisticated graph-based alternatives, surpassing the performances of Cai and Lam (2020). In particular, similarly to our work, Xu et al (2020a) leverage a multilingual framework inspired by Johnson et al (2017) and explore the possibility of pre-training on a range of related tasks, including MT; however, their focus is limited to showing the effectiveness of transfer learning from related tasks to English AMR parsing.…”

“…We report the Smatch and fine-grained scores that SGL and its current state-of-the-art alternatives attain on AMR-2.0 in Xu et al (2020a). Considering the similarity between the two approaches, this difference is likely caused by the increased number of tasks our model is asked to handle.…”

MRP 2020: The Second Shared Task on Cross-Framework and Cross-Lingual Meaning Representation Parsing

“…Since then, a new state-of-the-art has been established for English AMR, using sequenceto-sequence transduction (Zhang et al, 2019a,b) and iterative inference with graph encoding Lam, 2019, 2020). Xu et al (2020a) improved sequence-to-sequence parsing for AMR by using pre-trained encoders, reaching similar performance to Cai and Lam (2020). introduced a stack-transformer to enhance transitionbased AMR parsing (Ballesteros and Al-Onaizan, 2017), and Lee et al (2020) improved it further, using a trained parser for mining oracle actions and combining it with AMR-to-text generation to outperform the state of the-art.…”

AMR Parsing with Action-Pointer Transformer

“…Model Size In Table 3, we compare parameter sizes of recently published models alongside their parsing performances on AMR 2.0. Similar to our approach, most models use large pre-trained models to extract contextualized embeddings as fixed features, with the exception of Xu et al (2020), which is a seq-to-seq pre-training approach on large amount of data, and Bevilacqua et al (2021), which directly fine-tunes a seq-to-seq BART large (Lewis et al, 2019) model. 7 Except the large BART model, our APT small (3 layers) has the least number of trained parameters yet already surpasses all the previous models.…”