Investigating a neural language model’s replicability of psycholinguistic experiments: A case study of NPI licensing

Abstract

The recent success of deep learning neural language models such as Bidirectional Encoder Representations from Transformers (BERT) has brought innovations to computational language research. The present study explores the possibility of using a language model in investigating human language processes, based on the case study of negative polarity items (NPIs). We first conducted an experiment with BERT to examine whether the model successfully captures the hierarchical structural relationship between an NPI and its licensor and whether it may lead to an error analogous to the grammatical illusion shown in the psycholinguistic experiment (Experiment 1). We also investigated whether the language model can capture the fine-grained semantic properties of NPI licensors and discriminate their subtle differences on the scale of licensing strengths (Experiment 2). The results of the two experiments suggest that overall, the neural language model is highly sensitive to both syntactic and semantic constraints in NPI processing. The model’s processing patterns and sensitivities are shown to be very close to humans, suggesting their role as a research tool or object in the study of language.

Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics

“…(4-c)) received significantly higher surprisals than the illusion sentence (e.g., (4-a)). This finding replicates Shin et al (2023) in that, for the illusion condition ((4-a)) where no linearly precedes ever but is in an unlicensing position, ever incurs higher surprisal. It is interesting to see the sharp discrepancy between surprisal and perplexity, which we leave to Section 7.4 for discussion.…”

“…ever, any) to be acceptable, it has to be in the scope of negation. 15 Existing computational research has shown that the syntactic dependency between the licensor and the NPI is captured by language models (Jumelet and Hupkes, 2018;Jumelet et al, 2021;Shin et al, 2023) but with more difficulty as compared to subject-verb agreement or other syntactic dependencies (Marvin and Linzen, 2018;Warstadt et al, 2019Warstadt et al, , 2020. In this task, we expanded the suite of LMs and metrics and explored sensitivities to four types of licensors.…”

“…On the other hand, the discrepancy between sentence perplexity and surprisal makes it difficult to conclude to what degree LMs and humans overlap (cf. Shin et al, 2023).…”

How do linguistic illusions arise? Rational inference and good-enough processing as competing latent processes within individuals

“…In this context, it will likely be fruitful to take into account how state-of-the-art language and dialogue models like GPT and ChatGPT process illusion sentences, as they are purely data-driven: language models neither have an explicit error correction mechanism, nor do they have a reservoir of motivation and/or attention that can be depleted, unlike humans. There is already some data showing how language models differ (or not) from humans in the domain of linguistic illusions (e.g., Dentella et al, 2023;Cai et al, 2023;Shin et al, 2023;Paape, 2023;Zhang et al, 2023a), which can inform future investigations into the mechanisms that may be unique to human sentence processing.…”