Human Sentence Processing: Recurrence or Attention?

Merkx, Danny; Frank, Stefan

doi:10.18653/v1/2021.cmcl-1.2

Cited by 67 publications

(65 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They find that as long as the three types of models achieve a similar level of language modeling performance, there is no reliable difference in their predictive power. Merkx and Frank (2021) extend this study by comparing Transformer models against GRU models following similar experimental methods. The Transformer models are found to outperform the GRU models on explaining self-paced reading times and N400 measures but not eye-gaze durations.…”

Section: Related Workmentioning

confidence: 97%

Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators

Clark

Schuler

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Expectation-based theories of sentence processing posit that processing difficulty is determined by predictability in context. While predictability quantified via surprisal has gained empirical support, this representation-agnostic measure leaves open the question of how to best approximate the human comprehender's latent probability model. This article first describes an incremental left-corner parser that incorporates information about common linguistic abstractions such as syntactic categories, predicate-argument structure, and morphological rules as a computational-level model of sentence processing. The article then evaluates a variety of structural parsers and deep neural language models as cognitive models of sentence processing by comparing the predictive power of their surprisal estimates on self-paced reading, eye-tracking, and fMRI data collected during real-time language processing. The results show that surprisal estimates from the proposed left-corner processing model deliver comparable and often superior fits to self-paced reading and eye-tracking data when compared to those from neural language models trained on much more data. This may suggest that the strong linguistic generalizations made by the proposed processing model may help predict humanlike processing costs that manifest in latency-based measures, even when the amount of training data is limited. Additionally, experiments using Transformer-based language models sharing the same primary architecture and training data show a surprising negative correlation between parameter count and fit to self-paced reading and eye-tracking data. These findings suggest that large-scale neural language models are making weaker generalizations based on patterns of lexical items rather than stronger, more humanlike generalizations based on linguistic structure.

show abstract

Section: Related Workmentioning

confidence: 97%

Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators

Clark

Schuler

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Common optimization tasks for pretraining transformers, such as the masked LM task (Devlin et al, 2018) are quite similar to the word prediction tasks that are known to predict children's performance on other linguistic skills (Borovsky et al, 2012;Neuman et al, 2011;Gambi et al, 2020). Finally, TLMs tend to outperform other LMs in recent work modeling human reading times, eye-tracking data, and other psychological and psycholinguistic phenomena (Merkx and Frank, 2021;Schrimpf et al, 2020b,a;Hao et al, 2020;Bhatia and Richie, 2020;.…”

Section: Related Workmentioning

confidence: 98%

Can Transformer Language Models Predict Psychometric Properties?

Laverghetta¹,

Nighojkar²,

Mirzakhalov³

et al. 2021

Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics

View full text Add to dashboard Cite

Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired "commonsense" competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformerand non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformerbased LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning. 1

show abstract

“…Transformer architecture (Vaswani et al, 2017) has advanced the state of the art in a wide range of natural language processing (NLP) tasks (Devlin et al, 2019;Liu et al, 2019;. Along with this, Transformers have become a major subject of research from the viewpoints of engineering (Rogers et al, 2020) and scientific studies (Merkx and Frank, 2021;Manning et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models

Kobayashi¹,

Kuribayashi²,

Yokoi³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Transformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively analyzed. However, the Transformer architecture is not only composed of the multihead attention; other components can also contribute to Transformers' progressive performance. In this study, we extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization. Our analysis of Transformer-based masked language models shows that the token-to-token interaction performed via attention has less impact on the intermediate representations than previously assumed. These results provide new intuitive explanations of existing reports; for example, discarding the learned attention patterns tends not to adversely affect the performance. The codes of our experiments are publicly available. 1

show abstract

Human Sentence Processing: Recurrence or Attention?

Cited by 67 publications

References 35 publications

Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators

Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators

Can Transformer Language Models Predict Psychometric Properties?

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models

Contact Info

Product

Resources

About