Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge

Abstract

The question of whether humans represent grammatical knowledge as a binary condition on membership in a set of well-formed sentences, or as a probabilistic property has been the subject of debate among linguists, psychologists, and cognitive scientists for many decades. Acceptability judgments present a serious problem for both classical binary and probabilistic theories of grammaticality. These judgements are gradient in nature, and so cannot be directly accommodated in a binary formal grammar. However, it is also not possible to simply reduce acceptability to probability. The acceptability of a sentence is not the same as the likelihood of its occurrence, which is, in part, determined by factors like sentence length and lexical frequency. In this paper, we present the results of a set of large-scale experiments using crowd-sourced acceptability judgments that demonstrate gradience to be a pervasive feature in acceptability judgments. We then show how one can predict acceptability judgments on the basis of probability by augmenting probabilistic language models with an acceptability measure. This is a function that normalizes probability values to eliminate the confounding factors of length and lexical frequency. We describe a sequence of modeling experiments with unsupervised language models drawn from state-of-the-art machine learning methods in natural language processing. Several of these models achieve very encouraging levels of accuracy in the acceptability prediction task, as measured by the correlation between the acceptability measure scores and mean human acceptability values. We consider the relevance of these results to the debate on the nature of grammatical competence, and we argue that they support the view that linguistic knowledge can be intrinsically probabilistic.

Individual Differences in Language Acquisition and Processing

“…There has been considerable recent debate regarding whether grammatical judgments are categorical or probabilistic [118][119][120][121][122]. As in the acquisition example above, the two explanations predict different patterns of IDs.…”

SNAP judgments: A small N acceptability paradigm (SNAP) for linguistic acceptability judgments

“…In recent years, increases in the availability of data have led to further discussions of the weaknesses of the informal method (e.g. Arppe & Järvikivi 2007, Cowart 1997, Featherston 2005, Gibson & Fedorenko 2010, Gross & Culbertson 2011, Lau et al 2017, Linzen & Oseki 2015, Schütze 1996, Sorace & Keller 2005, Wasow & Arnold 2005. Such weaknesses include potential cognitive biases on the part of the researcher and participants, difficulty in controlling for discourse context, the inability to find interactions among factors, and the inability to find probabilistic effects or relative effect sizes.…”

Acceptability Rating of Ungrammatical Colloquial Latvian: How Native Speakers Judge Different Error Types

“…An important question from the perspective of computability of natural language: what are the relations between acceptability conducted by human raters and probability computed by a deterministic device (where the likelihood of occurrence is determined by factors such as sentence length and lexical frequency). Lau et al (2016) argues that although it is not simply possible to reduce acceptability to probability, acceptability can be predicted based on probability. A crucial generalization from the work by Lau et al (2016) indicates that linguistic knowledge is to a significant extent probabilistic.…”

“…Lau et al (2016) argues that although it is not simply possible to reduce acceptability to probability, acceptability can be predicted based on probability. A crucial generalization from the work by Lau et al (2016) indicates that linguistic knowledge is to a significant extent probabilistic. (Cp.…”