Can Transformer Language Models Predict Psychometric Properties?

Abstract

Transformer-based language models (LMs) continue to advance state-of-the-art performance on NLP benchmark tasks, including tasks designed to mimic human-inspired "commonsense" competencies. To better understand the degree to which LMs can be said to have certain linguistic reasoning skills, researchers are beginning to adapt the tools and concepts of the field of psychometrics. But to what extent can the benefits flow in the other direction? I.e., can LMs be of use in predicting what the psychometric properties of test items will be when those items are given to human participants? We gather responses from numerous human participants and LMs (transformerand non-transformer-based) on a broad diagnostic test of linguistic competencies. We then use the responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately. We then determine how well these two sets of predictions match. We find cases in which transformerbased LMs predict psychometric properties consistently well in certain categories but consistently poorly in others, thus providing new insights into fundamental similarities and differences between human and LM reasoning. 1

Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding

“…In particular, we hope to encourage the the development of new datasets that cover richer types of linguistic phenomena and language models that can learn essential linguistic skills that are generalizable. For future work, we plan to add more datasets to cover more phenomena such as psycholinguistics (Laverghetta Jr. et al, 2021). We envision our benchmark to be dynamic, meaning that a higher-quality and more difficult dataset for a phenomenon should replace the current ones in the future.…”

Developmental Negation Processing in Transformer Language Models

“…We train our models on N LI train for 10 epochs, using a learning rate of 1e − 5, a weight decay of 0.01, a batch size of 16, and a maximum sequence length 175. 3 We selected these hyperparameters to be similar to those which were previously reported to yield strong results when training on NLI datasets (Laverghetta Jr. et al, 2021). We additionally evaluated the models on N LI dev , and found that they all achieved a Matthews Correlation of at least 0.6 (Matthews, 1975), and thus concluded that these hyperparameters were suitable.…”

Developmental Negation Processing in Transformer Language Models

“…We train our models on N LI train for 10 epochs, using a learning rate of 1e − 5, a weight decay of 0.01, a batch size of 16, and a maximum sequence length 175. 3 We selected these hyperparameters to be similar to those which were previously reported to yield strong results when training on NLI datasets (Laverghetta Jr. et al, 2021). We additionally evaluated the models on N LI dev , and found that they all achieved a Correlation of at least 0.6 (Matthews, 1975), and thus concluded that these hyperparameters were suitable.…”

“…Negation is an important construct in language for reasoning over the truth of propositions (Heinemann, 2015), garnering interest from philosophy (Horn, 1989), psycholinguistics (Zwaan, 2012), and natural language processing (NLP) (Morante and Blanco, 2020). While transformer language models (TLMs) (Vaswani et al, 2017) have achieved impressive performance across many NLP tasks, a great deal of recent work has found that they do not process negation well, and often make predictions that would be trivially false in the eyes of a human (Rogers et al, 2020;Ettinger, 2020;Laverghetta Jr. et al, 2021).…”