Parsing All: Syntax and Semantics, Dependencies and Spans

Abstract

Both syntactic and semantic structures are key linguistic contextual clues, in which parsing the latter has been well shown beneficial from parsing the former. However, few works ever made an attempt to let semantic parsing help syntactic parsing. As linguistic representation formalisms, both syntax and semantics may be represented in either span (constituent/phrase) or dependency, on both of which joint learning was also seldom explored. In this paper, we propose a novel joint model of syntactic and semantic parsing on both span and dependency representations, which incorporates syntactic information effectively in the encoder of neural network and benefits from two representation formalisms in a uniform way. The experiments show that semantics and syntax can benefit each other by optimizing joint objectives. Our single model achieves new state-of-the-art or competitive results on both span and dependency semantic parsing on Propbank benchmarks and both dependency and constituent syntactic parsing on Penn Treebank.

LIMIT-BERT : Linguistics Informed Multi-Task BERT

“…Thus we employ semi-supervised learning to alleviate such data unbalance on multi-task learning by using a pre-trained linguistics model to label BooksCorpus and English Wikipedia data. The pre-trained model jointly learns POS tags and the four types of structures on semantics and syntax, in which the latter is from the XLNet version of (Zhou et al, 2020), giving state-of-the-art or comparable performance for the concerned four parsing tasks. During training, we set 10% probability to use gold syntactic parsing and SRL data: Penn Treebank (PTB) (Marcus et al, 1993), span style SRL (CoNLL-2005) (Carreras and Màrquez, 2005) and dependency style SRL (CoNLL-2009) (Hajič et al, 2009).…”

“…Firstly, we rebuild word representations from the WordPiece tokens for linguistics tasks. Then we follow (Zhou et al, 2020) to construct the taskspecific layers, including scoring layer and decoder layer. The former scores three types of linguistic objectives, dependency head, syntactic constituent and semantic role.…”

“…After the word-level construction, we calculate the POS tag, syntactic constituent, dependency head, and semantic role scores, following the training way as (Zhou et al, 2020) to construct syntactic constituent, dependency head, and semantic role scores objective loss which are represented as J 1 (θ), J 2 (θ) and J 3 (θ) respectively.…”

“…For span and dependency SRL, we use a single dynamic programming decoder according to the uniform semantic role score following the nonoverlapping constraints: span semantic arguments for the same predicate do not overlap (Punyakanok et al, 2008). For further details of the scoring and decoder layer, please refer to (Zhou et al, 2020).…”

“…Recently, pre-trained language models have shown greatly effective across a range of linguistics inspired natural language processing (NLP) tasks such as syntactic parsing, semantic parsing and so on Zhou et al, 2020;Ouchi et al, 2018;, when taking the latter as downstream tasks for the former. In the meantime, introducing linguistic clues such as syntax and semantics into the pre-trained language models may furthermore enhance other downstream tasks such as various Natural Language Understanding (NLU) tasks (Zhang et al, 2020a,b).…”