Incremental Processing of Principle B: Mismatches Between Neural Models and Humans

Abstract

Despite neural language models qualitatively capturing many human linguistic behaviors, recent work has demonstrated that they underestimate the true processing costs of ungrammatical structures. We extend these more fine-grained comparisons between humans and models by investigating the interaction between Principle B and coreference processing. While humans use Principle B to block certain structural positions from affecting their incremental processing, we find that GPT-based language models are influenced by ungrammatical positions. We conclude by relating the mismatch between neural models and humans to properties of training data and suggest that certain aspects of human processing behavior do not directly follow from linguistic data.Paper at CoNLL 2022

What language models can tell us about learning adjectives

“…Despite these apparent successes, there are noted limitations. For example, neural models struggle with infrequent constructions (e.g., Wei et al 2021) and in resolving conflicting linguistic processes (e.g., Davis 2022).…”

“…In linking models to acquisition, the claim is that grammatical knowledge follows directly from distributions observed in corpora. At least two, interrelated, components are necessary for this claim to hold: (i) linguistic data should contain the distributions corresponding to grammatical knowledge, and (ii) neural models should be able to recover these facts (see the discussion of superficialism in Davis (2022)).…”

“…Research on language models has largely investigated particular rules (e.g., agreement, coreference) that are robustly attested in corpora and/or are extremely regularized. However, we know that the accuracy of language models decreases as we move from frequently-attested and robust phenomena to more nuanced properties of the grammar (e.g., Davis 2022). Thus, in order to determine what they are actually learning, we argue that more "ambiguous" data should be considered -specifically, surface strings that could support different interpretations -given that the child has to learn a range of constructions and rules for which the availability of evidence varies in terms of its frequency or informativeness.…”