Large-scale benchmark yields no evidence that language model surprisal explains syntactic disambiguation difficulty

Huang, Kuan-Jung; Arehalli, Suhas; Kugemoto, Mari; Muxica, Christian; Prasad, Grusha; Dillon, Brian; Linzen, Tal

doi:10.1016/j.jml.2024.104510

Cited by 5 publications

(3 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 ), we found the inverse when classifying subtypes of aphasia (Table 4 ). This leads us to critically examine the view that larger size LLMs can be superior to their smaller counterparts 47 , 76 – 78 . It is worth reconsidering the supremacy of larger LLMs.…”

Section: Discussionmentioning

confidence: 99%

“…This is possibly because next word prediction within a sentence, a pre-training task shared by all the LLMs in our experiments, is not sufficient to capture the complex and subtle linguistic patterns in aphasia. LLMs-surprisals can complement the existing language features 47 .…”

Section: Discussionmentioning

confidence: 99%

“…Computational linguistic studies show that LLMs-surprisals is a valid predictor of human real-time sequence-by-sequence processing times 26 , 31 – 39 , and it has been used in (psycho-)linguistic 40 – 45 and morphosyntactic analyses 46 . On the other hand, Huang et al 47 and Amouyal et al 48 suggest that LLMs-surprisals cannot fully account for syntactic disambiguation difficulty and plausibility. Motivated by previous investigations in aphasia, NLP, and linguistics, we use LLMs surprisals to bridge LLMs and aphasia language analysis.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Clinical efficacy of pre-trained large language models through the lens of aphasia

Cong,

LaCroix,

Lee

2024

Sci Rep

View full text Add to dashboard Cite

The rapid development of large language models (LLMs) motivates us to explore how such state-of-the-art natural language processing systems can inform aphasia research. What kind of language indices can we derive from a pre-trained LLM? How do they differ from or relate to the existing language features in aphasia? To what extent can LLMs serve as an interpretable and effective diagnostic and measurement tool in a clinical context? To investigate these questions, we constructed predictive and correlational models, which utilize mean surprisals from LLMs as predictor variables. Using AphasiaBank archived data, we validated our models’ efficacy in aphasia diagnosis, measurement, and prediction. Our finding is that LLMs-surprisals can effectively detect the presence of aphasia and different natures of the disorder, LLMs in conjunction with the existing language indices improve models’ efficacy in subtyping aphasia, and LLMs-surprisals can capture common agrammatic deficits at both word and sentence level. Overall, LLMs have potential to advance automatic and precise aphasia prediction. A natural language processing pipeline can be greatly benefitted from integrating LLMs, enabling us to refine models of existing language disorders, such as aphasia.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%