SemEval-2016 Task 8: Meaning Representation Parsing

Abstract

In this report we summarize the results of the SemEval 2016 Task 8: Meaning Representation Parsing. Participants were asked to generate Abstract Meaning Representation (AMR) (Banarescu et al., 2013) graphs for a set of English sentences in the news and discussion forum domains. Eleven sites submitted valid systems. The availability of state-of-the-art baseline systems was a key factor in lowering the bar to entry; many submissions relied on CAMR (Wang et al., 2015b; Wang et al., 2015a) as a baseline system and added extensions to it to improve scores. The evaluation set was quite difficult to parse, particularly due to creative approaches to word representation in the web forum portion. The top scoring systems scored 0.62 F1 according to the Smatch (Cai and Knight, 2013) evaluation heuristic. We show some sample sentences along with a comparison of system parses and perform quantitative ablative studies.

ÚFAL-Oslo at MRP 2019: Garage Sale Semantic Parsing

“…Our results for DM and PSD are quite close to the original results reported in (Peng et al, 2017). Original SMATCH scores are reported in (May, 2016). The score reported on the LPPS subset is close to the original score, whereas the score measured on the whole test set is much lower.…”

Abstract Syntax as Interlingua: Scaling Up the Grammatical Framework from Controlled Languages to Robust Pipelines

“…The raw data for the synsets comes from existing resources such as Princeton WordNet for English (Fellbaum 1998), Svenska OrdNät (Viberg et al 2002) and WordNet-SALDO Following a similar task in text-to-AMR parsing (May 2016), a recent shared task at SemEval 2017 unveiled the state-of-the-art in AMR-to-text generation (May and Priyadarshi 2017). According to the SemEval 2017 Task 9 evaluation, the convincingly best-performing AMR-to-text generation system (Gruzitis, Gosko, and Barzdins 2017), among the contestants, combines a GF-based generator with the JAMR generator (Flanigan et al 2016), achieving the Trueskill score (human evaluation) of 1.03-1.07 and the BLEU score (automatic evaluation) of 18.82 (May and Priyadarshi 2017).…”

TUPA at MRP 2019: A Multi-Task Baseline System

“…Being a general DAG parser, TUPA has been shown (Hershcovich et al, 2018a,b) to support other graph-based meaning representations and similar frameworks, including UD (Universal Dependencies; Nivre et al, 2019), which was the focus of CoNLL 2017 and 2018 Shared Tasks (Zeman et al, 2017(Zeman et al, , 2018; AMR (Abstract Meaning Representation; Banarescu et al, 2013), targeted in SemEval 2016 and 2017 Shared Tasks (May, 2016;May and Priyadarshi, 2017); and DM (DELPH-IN MRS Bi-Lexical Dependencies; Ivanova et al, 2012), one of the target representations, among PAS and PSD (Prague Semantic Dependencies; Hajic et al, 2012;, in the SemEval 2014 and 2015 Shared Tasks on SDP (Semantic Dependency Parsing; Oepen et al, , 2015Oepen et al, , 2016. DM is converted from DeepBank , a corpus of hand-corrected parses from LinGO ERG (Copestake and Flickinger, 2000), an HPSG (Pollard and Sag, 1994) using Minimal Recursion Semantics (Copestake et al, 2005).…”