Rethinking Self-Attention: Towards Interpretability in Neural Parsing

Abstract

Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.

Automated Concatenation of Embeddings for Structured Prediction

“…We use the same setting as in Section 4. Mrini et al (2020) is not identical to the dataset in previous work such as Zhang et al (2020) and Wang and Tu (2020). ‡ : For reference, we confirmed with the authors of He and Choi (2020) that they used a different data pre-processing script with previous work.…”

UnImplicit Shared Task Report: Detecting Clarification Requirements in Instructional Text

“…We extract instances of clarifications from a resource of revision edits called wikiHowToImprove . Specifically, we used a state-of-the-art a constituency parser (Mrini et al, 2020) to preprocess all revisions from wikiHow- ToImprove and applied a set of rule-based filters to identify specific types of edits (see Table 2).…”

Recursive Tree-Structured Self-Attention for Answer Sentence Selection

“…several times alredy i found my self driving in the middle of the crossing in red light luckily at the moment no fines. hehehe :) pykester bank (PTB) (Marcus et al, 1994), and the Englishlanguage parser of Mrini et al (2020), which is the state of the art on the parse trees of the PTB.…”