Assessing introspective linguistic judgments quantitatively: the case of The Syntax of Chinese

Abstract

The informal judgments of the well-formedness of phrases and sentences have long been used as the primary data source for syntacticians. In recent years, the reliability of data based on linguists' introspective intuitions is increasingly subject to scrutiny. Although a number of studies were able to replicate a vast majority of English judgments published in a textbook and in peer-reviewed journal articles, the status of data in many non-English languages has yet to be experimentally examined. In this work, we employed formal quantitative methods to evaluate the reliability of judgments in the widely used textbook, The Syntax of Chinese (Huang, Li, & Li, 2009). We first assessed example sentences based on the acceptability ratings from 148 native Mandarin Chinese speakers. Using a target forced-choice task, we further explored the potentially problematic sentence pairs. Results of the two experiments suggest an eminently successful replication of judgments in the book: out of the 557 data samples tested, only five sentence pairs require further investigation. This large-scale study represents the first attempt to replicate the judgments in a non-English syntax textbook, in hopes to bridge the gap between the informal data-collection in Chinese linguistic research and the protocols of experimental cognitive science.

Revisiting subjunctive obviation in French: a formal acceptability judgment study

“…Some studies affirm the reliability of informal methods and highlight the importance of individual linguists' judgments (e.g. Phillips & Lasnik 2003;Featherston 2009;Phillips 2010;Sprouse et al 2013;Chen et al 2020), while other studies deny, doubt or question that reliability (Langendoen et al 1973;Schütze 2016Schütze [1996; Edelman & Christiansen 2003;Gibson & Fedorenko 2010;Gibson et al 2013;Linzen & Oseki 2018).…”

Raising the Bar on Acceptability Judgments Classification: An Experiment on ItaCoLA Using ELECTRA

“…It is important to note, however, that since acceptability has always fascinated scholars, small datasets had already been released before CoLa, mainly focused on theoretical linguistics or cognitive science-related tasks [41][42][43]. In addition to English, informal acceptability judgments have been evaluated in Hebrew and Japanese [32], as well as in French [44] and Chinese [45]. A small Italian dataset focusing on complexity and acceptability has also been released [46].…”