Predicting Human Psychometric Properties Using Computational Language Models

Laverghetta, Antonio; Nighojkar, Animesh; Mirzakhalov, Jamshidbek; Licato, John

doi:10.1007/978-3-031-04572-1_12

Cited by 12 publications

(13 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The interpretation of our results relies on the assumption that more accurate LMs provide better estimators of human surprisal, at least for those words which drive the superlinear fit of our GAMs. As discussed above, this assumption is supported by recent literature (Goodkind & Bicknell, 2018 ; Hao et al, 2020 ; Laverghetta et al, 2022 ; Merkx & Frank, 2021 ; Wilcox et al, 2020 ). Very recently, however, another line of work has emerged arguing that, to the contrary, lower perplexity LMs sometimes provide poorer fits to psychometric data.…”

Section: Discussionsupporting

confidence: 63%

“…While questions remain about the similarity between even the best modern LM’s predictions and those of humans, numerous studies in this area have found that higher quality LMs (those better able to predict test data) make better predictors of processing difficulty (Frank, 2009 ; Fossum & Levy, 2012 ; Goodkind & Bicknell, 2018 ; Wilcox et al, 2020 ). 7 Additionally, recent work comparing architectures has found that surprisal estimates from Transformer-based LMs (Vaswani et al, 2017 ) tend to be the best predictors of psychometric measures (Hao et al, 2020 ; Merkx & Frank, 2021 ; Laverghetta et al, 2022 ). 8 Only one recent published study—(Wilcox et al, 2020 )—has fit nonlinear GAMs of the linking function using surprisals from a modern Transformer-based LM (GPT-2 Radford et al, 2019 ).…”

Section: Surprisal Theorymentioning

confidence: 99%

See 1 more Smart Citation

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

et al. 2023

View full text Add to dashboard Cite

Words that are more surprising given context take longer to process. However, no incremental parsing algorithm has been shown to directly predict this phenomenon. In this work, we focus on a class of algorithms whose runtime does naturally scale in surprisal—those that involve repeatedly sampling from the prior. Our first contribution is to show that simple examples of such algorithms predict runtime to increase superlinearly with surprisal, and also predict variance in runtime to increase. These two predictions stand in contrast with literature on surprisal theory [Hale, 2001; Levy, 2008a], which assumes that the expected processing cost increases linearly with surprisal, and makes no prediction about variance. In the second part of this paper, we conduct an empirical study of the relationship between surprisal and reading time, using a collection of modern language models to estimate surprisal. We find that with better language models, reading time increases superlinearly in surprisal, and also that variance increases. These results are consistent with the predictions of sampling-based algorithms.

show abstract

Section: Discussionsupporting

confidence: 63%

Section: Surprisal Theorymentioning

confidence: 99%

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

et al. 2023

View full text Add to dashboard Cite

show abstract

“…A number of studies have observed that higher quality language models (those better able to predict test data) make better predictors of processing difficulty (Frank, 2009;Fossum and Levy, 2012;Goodkind and Bicknell, 2018;Wilcox et al, 2020). Additionally, recent work comparing LM architectures has found that transformer-based LMs (Vaswani et al, 2017) tend to be the best predictors of psychometric measures (Hao et al, 2020;Merkx and Frank, 2021;Laverghetta et al, 2022). 7 Only one recent study-Wilcox et al ( 2020)-has fit nonlinear models of the linking function using surprisals from a modern transformer-based LM (GPT-2 Radford et al, 2019).…”

Section: Empirical Studies In Surprisal Theorymentioning

confidence: 99%

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

Hoover¹,

Sonderegger²,

Piantadosi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Words that are more surprising given context take longer to process. However, no incremental parsing algorithm has been shown to directly predict this phenomenon. In this work, we focus on a class of algorithms whose runtime does naturally scale in surprisal---sampling algorithms. Our first contribution is to show that simple examples of such algorithms predict runtime to increase superlinearly with surprisal, and also predict variance in runtime to increase. These two predictions stand in contrast with literature on surprisal theory (Hale, 2001; Levy, 2008), which assumes that the expected processing cost increases linearly with surprisal, and makes no prediction about variance. In the second part of this paper, we conduct an empirical study of the relationship between surprisal and reading time, using a collection of modern language models to estimate surprisal. We find that with better language models, reading time increases superlinearly in surprisal, and also that variance increases. These results are consistent with the predictions of sampling-based algorithms.

show abstract

“…For instance, earlier generation automated item generation (AIG) tools (e.g., Gierl & Haladyna, 2012) required test developers to construct item blueprints with articulated factors from prespecified cognitive models. Large language models (LLMs), part of the ML toolkit, can assist test developers generate hundreds of thousands of items quickly, at scale, and with a large degree of variability (e.g., Belzak et al., 2023; Bezirhan & von Davier, 2023; Hommel et al., 2021; Laverghetta & Licato, 2023; von Davier, 2018). These items can be administered to candidates on large‐scale standardized tests, leading to high‐dimensional datasets with large degrees of sparsity across items of various response formats.…”

Section: Figurementioning

confidence: 99%

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

Zheng,

Nydick,

Huang

et al. 2024

Educational Measurement

View full text Add to dashboard Cite

The recent surge of machine learning (ML) has impacted many disciplines, including educational and psychological measurement (hereafter shortened as measurement). The measurement literature has seen rapid growth in applications of ML to solve measurement problems. However, as we emphasize in this article, it is imperative to critically examine the potential risks associated with involving ML in measurement. The MxML project aims to explore the relationship between measurement and ML, so as to identify and address the risks and better harness the power of ML to serve measurement missions. This paper describes the first study of the MxML project, in which we summarize the state of the field of applications, extensions, and discussions about ML in measurement contexts with a systematic review of the recent 10 years’ literature. We provide a snapshot of the literature in (1) areas of measurement where ML is discussed, (2) types of articles (e.g., applications, conceptual, etc.), (3) ML methods discussed, and (4) potential risks associated with involving ML in measurement, which result from the differences between what measurement tasks need versus what ML techniques can provide.

show abstract

Predicting Human Psychometric Properties Using Computational Language Models

Cited by 12 publications

References 32 publications

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

Contact Info

Product

Resources

About