Do We Need Neural Models to Explain Human Judgments of Acceptability?

Abstract

Native speakers can judge whether a sentence is an acceptable instance of their language. Acceptability provides a means of evaluating whether computational language models are processing language in a human-like manner. We test the ability of language models, simple language features, and word embeddings to predict native speakers’ judgments of acceptability on English essays written by non-native speakers. We find that much sentence acceptability variance can be captured by a combination of misspellings, word order, and word similarity (r = 0.494). While predictive neural models fit acceptability judgments well (r = 0.527), we find that a 4-gram model is just as good (r = 0.528). Thanks to incorporating misspellings, our 4-gram model surpasses both the previous unsupervised state-of-the art (r = 0.472), and the average native speaker (r = 0.46), demonstrating that acceptability is well captured by n-gram statistics and simple language features.

Are BERTs Sensitive to Native Interference in L2 Production?

“…Another explanation for the current lower performance for our embedding models relates to the knowledge structures in BERTs. Previously, studies indicated that for BERTs, the mid-layers have the best performance in syntactic-related tasks (Kelly et al, 2020). Such results may indicate that, unlike previous statistical models that reach the highest performance in output layers, embeddingbased models like BERTs may store different types of knowledge across different layers: the deeper the layers are, the more abstract concepts they will store.…”

“…Compared to state-of-the-art native language identifiers relying on statistical methods (such as Ngrams) with F-scores above 0.85, the performance of embedding models is still worse. Previous studies (Vajjala and Banerjee, 2017;Jing et al, 2020) argued that such worse predictions from embedding models indicate embedding models might have a stronger ability in capturing semantic information; while statistical-based language models might have better performance in capturing morphosyntactic and syntactic information.…”

“…With developments in computer science, data sciences, and cognitive and language sciences, artificial language models (LMs) based on language representations and embeddings have received attention in multiple areas. Current deep neural LMs based on various architectures can capture syntactic information through natural language input and utilize language knowledge to complete different tasks, such as identifying specific sentence structures (Enguehard et al, 2017;Kelly et al, 2020;Marvin and Linzen, 2018) and predicting grammar acceptability (Warstadt et al, 2019). Beyond training on a single language, multilingual neural language models could process cross-lingual tasks such as language identification and translation.…”

Exploring Linguistic Acceptability in Swedish Learners’ Language

“…There have been several studies on linguistic acceptability in English over the last years, using various forms of neural networks, targeting different error types, and focusing on different underlying aims. Neural networks trained to make acceptability judgements can yield for example theoretical insights into how language is perceived and acquired (Lawrence et al, 2000;Lau et al, 2017), or into what knowledge language models represent (Linzen et al, 2016;Jing et al, 2019). Practical applications of such models include evaluation of results from language-generating systems (such as question-answering or machine translation) or providing assistance in language learning.…”