Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Yu, Shaoyun; Gu, Chanyuan; Huang, Kexin; Li, Ping

doi:10.1126/sciadv.adn7744

Cited by 6 publications

(1 citation statement)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, there was a systematic association of concrete and abstract activation with different subdivisions of the DMN. Effects were most widespread in the left hemisphere though in some cases these were mirrored in the right, in line with other findings that conceptual processing in language tasks engages bilateral neural networks (Patel et al, 2023;Yu et al, 2024). 1 were supported by our analyses.…”

Section: Discussionsupporting

confidence: 92%

Neural specialisation for concrete and abstract concepts revealed through meta-analysis

Hoffman,

Bair

2024

Preprint

View full text Add to dashboard Cite

Identifying the brain systems that process concrete and abstract concepts is key to understanding the neural architecture of thought, memory and language. We review current theories of concreteness effects and test their neural predictions in a meta-analysis of 72 neuroimaging studies. Concrete concepts preferentially activated visual and action processing regions, particularly when presented in sentences, while abstract concepts preferentially activated networks for language, social cognition and semantic control. Specialisation for both concept types was present in the default mode network (DMN), with effects dissociating along a social-spatial axis. Concrete concepts generated greater activation in a medial temporal DMN component, implicated in constructing mental models of spatial contexts and scenes. Abstract concepts showed greater activation in frontotemporal DMN regions involved in theory-of-mind and language. These results support claims that generating models of situations and events is a core DMN function and indicate specialisation within DMN for different aspects of these models.

show abstract

Section: Discussionsupporting

confidence: 92%

Neural specialisation for concrete and abstract concepts revealed through meta-analysis

Hoffman,

Bair

2024

Preprint

View full text Add to dashboard Cite

show abstract

Non‐Autoregressive Translation Algorithm Based on LLM Knowledge Distillation in English Corpus

Ju,

Wang

2024

Engineering Reports

View full text Add to dashboard Cite

Although significant advancements have been made in the quality of machine translation by large‐scale language models, their high computational costs and resource consumption have hindered their widespread adoption in practical applications. So this research introduces an English corpus‐based machine translation algorithm that leverages knowledge distillation from large language model, with the goal of enhancing translation quality and reducing the computational demands of the model. Initially, we conducted a thorough analysis of the English corpus to identify prevalent language patterns and structures. Following this, we developed a knowledge distillation approach that transfers the translation expertise of a large teacher model to a smaller student model, thereby achieving increased translation accuracy and efficiency. We designed a dynamic temperature hyperparameter distillation strategy that effectively enhances the precision of translations. In the experimental phase, we utilized several standard English corpora to train and assess our algorithm. The findings indicate that, compared to current machine translation systems, our method significantly reduces the need for computational resources while preserving translation quality.

show abstract

Instruction-tuned large language models misalign with natural language comprehension in humans

Gao,

Ma,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

Transformer-based language models have significantly advanced our understanding of meaning representation in the human brain. Prior research utilizing smaller models like BERT and GPT-2 suggests that "next-word prediction" is a computational principle shared between machines and humans. However, recent advancements in large language models (LLMs) have highlighted the effectiveness of instruction tuning beyond next-word prediction. It remains to be tested whether instruction tuning can further align the model with language processing in the human brain. In this study, we evaluated the self-attention of base and finetuned LLMs of different sizes against human eye movement and functional magnetic resonance imaging (fMRI) activity patterns during naturalistic reading. Our results reveal that increases in model size significantly enhance the alignment between LLMs and brain activity, whereas instruction tuning does not. These findings confirm a scaling law in LLMs' brain-encoding performance and suggest that "instruction-following" may not mimic natural language comprehension in humans.

show abstract

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Cited by 6 publications

References 70 publications

Neural specialisation for concrete and abstract concepts revealed through meta-analysis

Neural specialisation for concrete and abstract concepts revealed through meta-analysis

Non‐Autoregressive Translation Algorithm Based on LLM Knowledge Distillation in English Corpus

Instruction-tuned large language models misalign with natural language comprehension in humans

Contact Info

Product

Resources

About