SNAP judgments: A small N acceptability paradigm (SNAP) for linguistic acceptability judgments

Abstract

While published linguistic judgments sometimes differ from the judgments found in large-scale formal experiments with naive participants, there is not a consensus as to how often these errors occur nor as to how often formal experiments should be used in syntax and semantics research. In this article, we first present the results of a large-scale replication of the Sprouse et al. 2013 study on 100 English contrasts randomly sampled from Linguistic Inquiry 2001-2010 and tested in both a forced-choice experiment and an acceptability rating experiment. Like Sprouse, Schütze, and Almeida, we find that the effect sizes of published linguistic acceptability judgments are not uniformly large or consistent but rather form a continuum from very large effects to small or nonexistent effects. We then use this data as a prior in a Bayesian framework to propose a small n acceptability paradigm for linguistic acceptability judgments (SNAP Judgments). This proposal makes it easier and cheaper to obtain meaningful quantitative data in syntax and semantics research. Specifically, for a contrast of linguistic interest for which a researcher is confident that sentence A is better than sentence B, we recommend that the researcher should obtain judgments from at least five unique participants, using at least five unique sentences of each type. If all participants in the sample agree that sentence A is better than sentence B, then the researcher can be confident that the result of a full forced-choice experiment would likely be 75% or more agreement in favor of sentence A (with a mean of 93%). We test this proposal by sampling from the existing data and find that it gives reliable performance.*

A Self‐Organizing Approach to Subject–Verb Number Agreement

“…Mahowald et al. () calculate the expected probability of unanimity in such a test with three participants and three items to be 0.89 (95% CI [0.7, 1.0]). Thus, we feel justified in having done only the reported pairwise comparisons to investigate the hypothesized feature hierarchy.…”

“…The three authors of this paper agreed that Containers were more acceptable than Measures Phrases and Quantifiers for the two Container-hood tests, that Collections were more acceptable than Measure Phrases and Quantifiers for the Spatial Configuration test, and that Measures were more acceptable than Collections for the Abstract N2 tests. Mahowald et al (2016) calculate the expected probability of unanimity in such a test with three participants and three items to be 0.89 (95% CI [0.7, 1.0]). Thus, we feel justified in having done only the reported pairwise comparisons to investigate the hypothesized feature hierarchy.…”

Individual Differences in Language Acquisition and Processing

“…There has been considerable recent debate regarding whether grammatical judgments are categorical or probabilistic [118][119][120][121][122]. As in the acquisition example above, the two explanations predict different patterns of IDs.…”

Quantifying sentence acceptability measures: Reliability, bias, and variability

“…For these reasons, the characteristics of different kinds of acceptability measures are well studied. We know that acceptability judgment data are influenced by details such as the selection of participants (Dąbrowska 2010), sample size (Mahowald et al 2016), task structure (Featherston 2008), participant engagement , and data processing decisions (Juzek 2015).…”