Large-scale evidence for logarithmic effects of word predictability on reading time

Shain, Cory; Meister, Clara; Pimentel, Tiago; Cotterell, Ryan; Levy, Roger

doi:10.1073/pnas.2307876121

Cited by 13 publications

(12 citation statements)

References 100 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our findings suggest that in subtyping aphasia, machine learning models involving larger scale LLMs underperform those involving smaller LLMs (Table 4 ). This result also indirectly aligns with Oh and Schuler 81 and Shain et al 38 . They found that larger scale LLMs show a worse fit to human reading times.…”

Section: Discussionsupporting

confidence: 90%

Clinical efficacy of pre-trained large language models through the lens of aphasia

Cong,

LaCroix,

Lee

2024

Sci Rep

View full text Add to dashboard Cite

The rapid development of large language models (LLMs) motivates us to explore how such state-of-the-art natural language processing systems can inform aphasia research. What kind of language indices can we derive from a pre-trained LLM? How do they differ from or relate to the existing language features in aphasia? To what extent can LLMs serve as an interpretable and effective diagnostic and measurement tool in a clinical context? To investigate these questions, we constructed predictive and correlational models, which utilize mean surprisals from LLMs as predictor variables. Using AphasiaBank archived data, we validated our models’ efficacy in aphasia diagnosis, measurement, and prediction. Our finding is that LLMs-surprisals can effectively detect the presence of aphasia and different natures of the disorder, LLMs in conjunction with the existing language indices improve models’ efficacy in subtyping aphasia, and LLMs-surprisals can capture common agrammatic deficits at both word and sentence level. Overall, LLMs have potential to advance automatic and precise aphasia prediction. A natural language processing pipeline can be greatly benefitted from integrating LLMs, enabling us to refine models of existing language disorders, such as aphasia.

show abstract

Section: Discussionsupporting

confidence: 90%

Clinical efficacy of pre-trained large language models through the lens of aphasia

Cong,

LaCroix,

Lee

2024

Sci Rep

View full text Add to dashboard Cite

show abstract

“…This is precisely the probability distribution we obtain from a language model after it has been given w 0 , …, w i −1 as input. The relationship between human reading times and surprisal estimated from a language model in this fashion has been found to be approximately linear (Shain et al, 2024 ; Smith & Levy, 2013 ).…”

Section: Methodsmentioning

confidence: 99%

Neural Networks as Cognitive Models of the Processing of Syntactic Constraints

Arehalli,

Linzen

2024

Open Mind

View full text Add to dashboard Cite

Languages are governed by syntactic constraints—structural rules that determine which sentences are grammatical in the language. In English, one such constraint is subject-verb agreement, which dictates that the number of a verb must match the number of its corresponding subject: “the dogs run”, but “the dog runs”. While this constraint appears to be simple, in practice speakers make agreement errors, particularly when a noun phrase near the verb differs in number from the subject (for example, a speaker might produce the ungrammatical sentence “the key to the cabinets are rusty”). This phenomenon, referred to as agreement attraction, is sensitive to a wide range of properties of the sentence; no single existing model is able to generate predictions for the wide variety of materials studied in the human experimental literature. We explore the viability of neural network language models—broad-coverage systems trained to predict the next word in a corpus—as a framework for addressing this limitation. We analyze the agreement errors made by Long Short-Term Memory (LSTM) networks and compare them to those of humans. The models successfully simulate certain results, such as the so-called number asymmetry and the difference between attraction strength in grammatical and ungrammatical sentences, but failed to simulate others, such as the effect of syntactic distance or notional (conceptual) number. We further evaluate networks trained with explicit syntactic supervision, and find that this form of supervision does not always lead to more human-like syntactic behavior. Finally, we show that the corpus used to train a network significantly affects the pattern of agreement errors produced by the network, and discuss the strengths and limitations of neural networks as a tool for understanding human syntactic processing.

show abstract

“…Frequency values were extracted from the SUBTLEX corpus of American film subtitles (Brysbaert & New, 2009), commonly used as a proxy for standard-English word frequency and which has been shown to correlate with reading-time behavior. Because the impact of word frequency on reading times is logarithmic as opposed to linear (Shain et al, 2024), we used the Zipf values (which are both logarithmic and standardized) as opposed to raw counts (Van Heuven et al, 2014). 3 To avoid including noncontent words, we limited our analysis of frequency to the words in our corpora marked as a verb, noun, adjective, or adverb according to Stanza.…”

Section: Word Frequencymentioning

confidence: 99%

So much for plain language: An analysis of the accessibility of U.S. federal laws over time.

Martínez,

Mollica,

Gibson

2024

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

Over the last 50 years, there have been efforts on behalf of the U.S. government to simplify legal documents for society at large. However, there has been no systematic evaluation of how effective these effortscollectively referred to as the "plain-language movement"-have been. Here we report the results of a large-scale longitudinal corpus analysis (n ≈ 225 million words), in which we compared every law passed by congress with a comparably sized sample of English texts from four different baseline genres published during approximately the same time period. We also compared the entirety of the U.S. Code (the official compilation of all federal legislation currently in force) with a large sample of recently published texts from six baseline genres of English. We found that laws remain laden with features associated with psycholinguistic complexity-including center-embedding, passive voice, low-frequency jargon, capitalization, and sentence length-relative to the baseline genres of English, and that the prevalence of most of these features has not meaningfully declined since the initial onset of the plain-language efforts. These findings suggest top-down efforts to simplify legal texts have thus far remained largely ineffectual, despite the apparent tractability of these changes, and call into question the coherence and legitimacy of legal doctrines whose validity rests on the notion of laws being easily interpretable by laypeople. Public Significance StatementOver the last 50 years, there have been efforts on behalf of the U.S. government to simplify legal documents for society at large. How effective have these efforts been? Here we analyzed every law passed by Congress between the years 1951 to 2022 along with a large sample of baseline texts published over roughly the same time period and found that laws remain laden with features associated with processing difficulty relative to other genres, and that the prevalence of most of these features has not meaningfully declined. We found similar results when comparing laws with academic texts. These findings suggest topdown efforts to simplify legal texts have thus far remained largely ineffectual, despite the apparent tractability of these changes, raising and informing questions of law and public policy.

show abstract

Large-scale evidence for logarithmic effects of word predictability on reading time

Cited by 13 publications

References 100 publications

Clinical efficacy of pre-trained large language models through the lens of aphasia

Clinical efficacy of pre-trained large language models through the lens of aphasia

Neural Networks as Cognitive Models of the Processing of Syntactic Constraints

So much for plain language: An analysis of the accessibility of U.S. federal laws over time.

Contact Info

Product

Resources

About