The need for quantitative methods in syntax and semantics research

Abstract

The prevalent method in syntax and semantics research involves obtaining a judgment of the acceptability of a sentence / meaning pair, typically by just the author of the paper, sometimes with feedback from colleagues. This methodology does not allow proper testing of scientific hypotheses because of (a) the small number of experimental participants (typically one); (b) the small number of experimental stimuli (typically one); (c) cognitive biases on the part of the researcher and participants; and (d) the effect of the preceding context (e.g., other constructions the researcher may have been recently considering). In the current paper we respond to some arguments that have been given in support of continuing to use the traditional non-quantitative method in syntax / semantics research. One recent defense of the traditional method comes from Phillips (2008), who argues that no harm has come from the non-quantitative approach in syntax research thus far. Phillips argues that there are no cases in the literature where an incorrect intuitive judgment has become the basis for a widely accepted generalization or an important theoretical claim. He therefore concludes that there is no reason to adopt more rigorous data collection standards. We challenge Philips' conclusion by presenting three cases from the literature where a faulty intuition has led to incorrect generalizations and mistaken theorizing, plausibly due to cognitive biases on the part of the researchers.Furthermore, we present additional arguments for rigorous data collection standards. For example, allowing lax data collection standards has the undesirable effect that the results and claims will often be ignored by researchers with stronger methodological standards.Finally, we observe that behavioral experiments are easier to conduct in English than ever before, with the advent of Amazon.com's Mechanical Turk, a marketplace interface that Quantitative syntax 3 can be used for collecting behavioral data over the internet.Quantitative syntax 4

A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010

“…Because informal methods, by their very nature, do not involve any reports of the data collection technique, it is difficult to engage in discussions of the "typical" data collection method. It is not uncommon for critics of informal methods to claim that linguists consult only one participant for judgments (the linguist herself), and use only one item per sentence type (the examples published in the journal article) (e.g., Gibson and Fedorenko 2013). In our experience, linguists tend to use many more participants and many more items than these reports suggest; however, it is an empirical question exactly how much variation there is in informal methods.…”

“…We believe that the first step in understanding the relative merits of each family of methods is to determine to what extent the two methods converge (or diverge). Although there have been several previous studies that have compared results of informal methods with the results of formal methods (e.g., Gordon and Hendrick, 1997;Clifton, Fanselow and Frazier, 2006;Gibson and Fedorenko, 2013), these previous studies cannot in principle be used to estimate a convergence rate between informal and formal methods, for two reasons. First, these studies have investigated a relatively small number of phenomena (e.g, Gibson and Fedorenko, 2013 test seven data points comprising three distinct phenomena in their criticism of informal methods) compared to the sheer number of data points published in the syntax literature.…”

“…For the present study we decided to focus exclusively on standard acceptability judgments for three reasons: (i) they are the most straightforward to adapt to formal methods, as they require no special instruction of the participants, and no special equipment for participants to complete the task; (ii) they form the largest single type of data published in LI 2001-2010; and (iii) they are the focus of several recent criticisms of informal methods in the literature (e.g., Ferreira 2005, Wasow and Arnold 2005, Gibson and Fedorenko 2010, Gibson and Fedorenko 2013. Of course, the fact that the current study is limited to a single data type means that the estimate of convergence derived here applies only to that data type.…”

“…This development means that there are two methods for collecting acceptability judgments currently in widespread use in the field of syntax: the relatively informal traditional methods that have largely established the foundation of the field for the past 60 years (henceforth informal methods), and the more formal experimental methods that have been gaining popularity over the past 15 years (henceforth formal methods). This methodological dichotomy has led a number of researchers to ask which method is empirically superior (e.g., Bard et al, 1996;Keller, 2000;Edelman and Christiansen, 2003;Phillips and Lasnik, 2003;Featherston, 2005aFeatherston, , 2005bFeatherston, , 2007Featherston, , 2008Featherston, , 2009Ferreira, 2005;Sorace and Keller, 2005;Wasow and Arnold, 2005;den Dikken et al, 2007;Alexopoulou and Keller, 2007;Bornkessel-Schlesewsky and Schlesewsky, 2007;Fanselow, 2007;Grewendorf, 2007;Haider, 2007;Newmeyer, 2007;Sprouse, 2007;Culbertson and Gross, 2009;Myers, 2009aMyers, , 2009bPhillips, 2010;Bader and Häussler, 2010;Dąbrowska, 2010;Gibson and Fedorenko, 2010;Culicover and Jackendoff, 2010;Gross and Culberton, 2011;Sprouse, 2011b;Weskott and Fanselow, 2011;Gibson et al, 2011;Almeida, 2012, 2013;Gibson and Fedorenko, 2013). Our goal in this paper is to substantially increase the empirical basis of this line of research by comparing the results of informal and formal methods for a very large and random s...…”

“…Furthermore, some critics of informal methods have suggested that the existence of such marketplaces reduces the time cost of formal experiments (e.g., Gibson and Fedorenko, 2013;Gibson et al 2011), therefore it seems appropriate to use these marketplaces for this case study.…”