Structural Supervision Improves Learning of Non-Local Grammatical Dependencies

Abstract

State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a question that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus two models that represent hierarchical structure and deploy it in left-to-right processing: Recurrent Neural Network Grammars (RNNGs) and a incrementalized version of the Parsing-as-Language-Modeling configuration from Charniak et al. (2016). Models are tested on a diverse range of configurations for two classes of non-local grammatical dependencies in English-Negative Polarity licensing and Filler-Gap Dependencies. Using the same training data across models, we find that structurally-supervised models outperform the LSTM, with the RNNG demonstrating best results on both types of grammatical dependencies and even learning many of the Island Constraints on the filler-gap dependency. Structural supervision thus provides data efficiency advantages over purely stringbased training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies.

Compound Probabilistic Context-Free Grammars for Grammar Induction

“…RNNGs are generative models of language that jointly model syntax and surface structure by incrementally generating a syntax tree and sentence. As with NLMs, RNNGs make no independence assumptions, and have been shown to outperform NLMs in terms of perplexity and grammaticality judgment when trained on gold trees (Kuncoro et al, 2018;Wilcox et al, 2019). We take the best run from each model and parse the training set, 15 and use the induced trees to supervise an RNNG for each model using the parameterization from Kim et al (2019).…”

Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations

“…Crucially, although islands block the fillers from associating with gaps within the island, they do not prohibit association between fillers and gaps that occur structurally to the right of the island, as shown in Figure 3. Wilcox et al (2019b) found that while large scale models are able to thread the 2 × 2 contingency between fillers and gaps into syntactically complex material-such as through numerous sentential embeddings-they do not thread the dependency into some island configurations. Inside of relative clauses and temporal adjuncts, for example, the presence or absence of an upstream filler has no effect on the relative surprisal of a gap, and the wh-licensing interaction drops to near zero.…”

A Systematic Assessment of Syntactic Generalization in Neural Language Models

“…• Long-Distance Dependencies are covariations between two tokens that span long distances in tree depth. Test suites include Filler-Gap Dependencies (FGD) (6 suites) from and Wilcox et al (2019b), and 2 novel Cleft suites, described in detail below. 1b)) and lexicalized verbs should set up expectations for NPs (the region in bold should have a lower surprisal in (1d) than in (1c)).…”