Transition-based Parsing with Stack-Transformers

Abstract

Modeling the parser state is key to good performance in transition-based parsing. Recurrent Neural Networks considerably improved the performance of transition-based systems by modelling the global state, e.g. stack-LSTM parsers, or local state modeling of contextualized features, e.g. Bi-LSTM parsers. Given the success of Transformer architectures in recent parsing systems, this work explores modifications of the sequence-to-sequence Transformer architecture to model either global or local parser states in transition-based parsing. We show that modifications of the cross attention mechanism of the Transformer considerably strengthen performance both on dependency and Abstract Meaning Representation (AMR) parsing tasks, particularly for smaller models or limited training data. * Miguel's and Austin's contributions were carried out while at IBM Research.

Probabilistic, Structure-Aware Algorithms for Improved Variety, Accuracy, and Coverage of AMR Alignments

“…Then, the alignments are "tuned" with a parser oracle to select the candidates that correspond to the oracle parse that is most similar to the gold AMR. Some AMR parsers (Naseem et al, 2019;Fernandez Astudillo et al, 2020) Figure 1: AMR and alignments for the sentence "Most of the students want to visit New York when they graduate." Alignments are differentiated by colors: blue (subgraphs), green (duplicate subgraphs), and orange (relations).…”

Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction

“…Given an input sentence S, we first use a pretrained transformer-based AMR parser (Fernandez Astudillo et al, 2020) to obtain the AMR graph for S. We then use RoBERTa to encode each sentence to identify entity mentions and event triggers as candidate nodes. After that, we map each candidate node to AMR nodes and enforce message passing using a GATbased semantic graph aggregator to capture global inter-dependency between candidate nodes.…”

“…We employ a transformer based AMR parser (Fernandez Astudillo et al, 2020) pre-trained on AMR 3.0 annotations 2 to generate an AMR graph G a = (V a , E a ) with an alignment between AMR nodes and word spans in an input sentence S. Each node v a i = (m a i , n a i ) ∈ V a represents an AMR concept or predicate, and we use m a i and n a i to denote the starting and ending indices of such a node in the original sentence. For AMR edges, we use e a i,j to denote the specific relation type between nodes v a i and v a j in AMR annotations.…”

Structural Guidance for Transformer Language Models

“…Finally, our approach relates to the other works that propose ways of incorporating structural information into Transformer-based models. This includes the use of dependency or tree structure for constraining self-attention patterns (Strubell et al, 2018;Wang et al, 2019;, guiding cross-attention (Chen et al, 2018;Astudillo et al, 2020), modelling syntactic distance (Du et al, 2020), using syntactic information to guide the computation flow in the model (Shen et al, 2021), or through knowledge distillation (Kuncoro et al, 2020). Our structured masking in parsing as language modeling approach is close in spirit to the methods that modify attention mechanism according to syntactic connections (Astudillo et al, 2020); This work, however, primarily aims to study the impact of structural guidance on syntactic generalization.…”