Maximum Bayes Smatch Ensemble Distillation for AMR Parsing

Abstract

AMR parsing has experienced an unprecendented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learning. Self-learning techniques have also played a role in pushing performance forward. However, for most recent high performant parsers, the effect of self-learning and silver data augmentation seems to be fading. In this paper we propose to overcome this diminishing returns of silver data by combining Smatch-based ensembling techniques with ensemble distillation. In an extensive experimental setup, we push single model English parser performance to a new state-of-the-art, 85.9 (AMR2.0) and 84.3 (AMR3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian and Spanish. Finally we explore the impact of the proposed technique on domain adaptation, and show that it can produce gains rivaling those of human annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR.

Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs

“…The use of a reference metric, such as Rouge-L, to ensemble the outputs of multiple language distributions is a common technique in Minimum Bayesian Risk decoding, with applications to speech-to-text (Goel and Byrne, 2000), machine translation (Kumar and Byrne, 2004), language modeling (Suzgun et al, 2022) and parsing (Lee et al, 2022), among others. Here we use a similar technique in the context of instruction generation.…”

“You Are An Expert Linguistic Annotator”: Limits of LLMs as Analyzers of Abstract Meaning Representation

“…These semantic parsing datasets, which focus on producing database queries in particular domains, are less complex and domain-general than AMR, but the results suggest that LLMs should contain aspects of the knowledge needed to analyze semantic structure. As for AMR, pre-trained transformer models have helped to advance the state of the art in AMR parsing, with recent AMR parsers building on the foundation of models like BART (Bevilacqua et al, 2021;Bai et al, 2022;Lee et al, 2022;Zhou et al, 2021). This indicates that pre-trained models may also pick up on representational capabilities relevant for supporting AMR.…”

AMRs Assemble! Learning to Ensemble with Autoregressive Models for AMR Parsing

“…We use two variants of Graphene, i) Graphene base , where every input graph is chosen as a pivot graph once, and the best among the modified pivot graphs is chosen as the final prediction based on average support; and ii) Graphene smatch , which is similar to Graphene base but chooses the best modified pivot graph based on average SMATCH score, similar to Barzdins and Gosko (2016). We do not compare our approach using Maximum Bayes SMATCH Ensemble (Lee et al, 2022), as it is a technique for producing high-quality silver data by combining SMATCH-based ensembling techniques with ensemble distillation, and its code and data are not publicly available.…”

“…Nevertheless, in an attempt to push SMATCH performance, there has been a recent trend towards ensemble models, which merge AMR graph predictions from multiple systems. Some examples include Graphene (Lam et al, 2021), a graph mining algorithm that searches for the largest common structure among the graph predictions, or the Maximum Bayes SMATCH Ensemble (Lee et al, 2022), which introduces a Bayesian ensemble approach in order to create high-quality silver data. However, notwithstanding their higher performance, ensem-ble models are potentially more vulnerable to producing corrupted AMR graphs.…”