Colorless green ideas do sleep furiously: gradient acceptability and the nature of the grammar

Abstract

In their recent paper, Lau, Clark, and Lappin explore the idea that the probability of the occurrence of word strings can form the basis of an adequate theory of grammar (Lau, Jey H., Alexander Clark & 15 Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A prob- abilistic view of linguistic knowledge. Cognitive Science 41(5):1201–1241). To make their case, they present the results of correlating the output of several probabilistic models trained solely on naturally occurring sentences with the gradient acceptability judgments that humans report for ungrammatical sentences derived from roundtrip machine translation errors. In this paper, we first explore the logic of the Lau et al. argument, both in terms of the choice of evaluation metric (gradient acceptability), and in the choice of test data set (machine translation errors on random sentences from a corpus). We then present our own series of studies intended to allow for a better comparison between LCL’s models and existing grammatical theories. We evaluate two of LCL’s probabilistic models (trigrams and recurrent neural network) against three data sets (taken from journal articles, a textbook, and Chomsky’s famous colorless-green-ideas sentence), using three evaluation metrics (LCL’s gradience metric, a categorical version of the metric, and the experimental-logic metric used in the syntax literature). Our results suggest there are very real, measurable cost-benefit tradeoffs inherent in LCL’s models across the three evaluation metrics. The gain in explanation of gradience (between 13% and 31% of gradience) is offset by losses in the other two metrics: a 43%-49% loss in coverage based on a categorical metric of explaining acceptability, and a loss of 12%-35% in explaining experimentally-defined phenomena. This suggests that anyone wishing to pursue LCL’s models as competitors with existing syntactic theories must either be satisfied with this tradeoff, or modify the models to capture the phenomena that are not currently captured.

Intralingual Variation in Acceptability Judgments and Production: Three Case Studies in Russian Grammar

“…A replication of this study was performed by Sprouse et al (2018). The major criticism of the results from Lau et al (2014Lau et al ( , 2015Lau et al ( , 2017 concerns the fact that round-trip translations might not create grammatical oppositions of the kind usually devised by syntacticians, whereby a specific grammatical property is manipulated while other properties remain constant in an experimental set.…”

“…Consequently, on the basis of probabilities found in a corpus, one ought to be able to predict acceptability judgments. To formalize the gathering of these probabilities, investigators used language models that were fitted to the annotated corpus data in a supervised (Bresnan, 2007) or unsupervised manner (Lau et al, 2017;Sprouse et al, 2018). Bresnan (2007) explored the correspondence between the two data sources with respect to the English dative alternation (e.g., give the boy the book vs. give the book to the boy).…”

“…It is also essential to consider what types of texts are represented in corpora. While Bresnan (2007) used the Switchboard corpus of spontaneous speech, in the study by Lau et al (2017) this factor was not controlled for, and Sprouse et al (2018) used utterances from research literature. The problem is that data from texts belonging to particular genres might not be comparable to the results of acceptability tasks in which speakers are asked to evaluate the naturalness of the stimuli.…”

“…This assumption is questionable as there is evidence that forced-choice is a form of rating task. Sprouse et al (2018) reports that the results of forced-choice tasks, when transformed into ratings by means of the Elo system first developed for rating the relative strength of chess players, in fact correspond directly to the results of Likert scale tasks. The results of the experiments by Verhoeven and Temme (2017) turned out to be highly correlated.…”

Modeling Hierarchical Syntactic Structures in Morphological Processing

“…Second, given that computational models are multiplicative in nature (Yang, 2017), it is not fair to simultaneously test the words with different numbers of morphemes. That is, shorter words are exponentially more probable than longer ones, but shorter expressions are not necessarily more acceptable or easier to process (Lau et al, 2016;Sprouse et al, 2018). Given these two reasons, for each MCCV iteration, 10% of the bimorphemic words (n = 655) was randomly held out as the testing data and the remaining 90% (n = 13,244 -655 = 12,589) was assigned as the train- (2007) for details.…”