Life after BERT: What do Other Muppets Understand about Language?

Abstract

Existing pre-trained transformer analysis works usually focus only on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives. In our work, we utilize the oLMpics benchmark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT. Additionally, we adapt the oLMpics zero-shot setup for autoregressive models and evaluate GPT networks of different sizes. Our findings show that none of these models can resolve compositional questions in a zero-shot fashion, suggesting that this skill is not learnable using existing pre-training objectives.Furthermore, we find that global model decisions such as architecture, directionality, size of the dataset, and pre-training objective are not predictive of a model's linguistic capabilities. The code is available on GitHub 1 .

Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning

“…Han et al (2021) and Liu et al (2022) prompted GPT3 to generate synthetic translation and NLI datasets, respectively. Lialin et al (2022) and Ettinger (2019) evaluated language models on smaller datasets for negation and role reversal. We extend these datasets to around 1500 data points and evaluate 22 models, including GPT3.…”

“…The field of analysis of pre-trained models has grown rapidly in recent years (Zagoury et al, 2021;Liu et al, 2021;Lialin et al, 2022;bench authors, 2023;Rogers et al, 2020). Methods such as attention pattern analysis (Kovaleva et al, 2019;Kobayashi et al, 2020), linear probing (Tenney et al, 2019), and zero-shot probing (Belinkov et al, 2020;Talmor et al, 2019;Ettinger, 2019;Lialin et al, 2022) allow us to evaluate specific capabilities of pre-trained models.…”

“…The field of analysis of pre-trained models has grown rapidly in recent years (Zagoury et al, 2021;Liu et al, 2021;Lialin et al, 2022;bench authors, 2023;Rogers et al, 2020). Methods such as attention pattern analysis (Kovaleva et al, 2019;Kobayashi et al, 2020), linear probing (Tenney et al, 2019), and zero-shot probing (Belinkov et al, 2020;Talmor et al, 2019;Ettinger, 2019;Lialin et al, 2022) allow us to evaluate specific capabilities of pre-trained models. Zero-shot methods give us arguably the most clear picture, as they directly probe what the model learned through the upstream task and allow the researcher to target very specific skills such as understanding of negation or role.…”

“…Psycholinguistic datasets used in a study by Ettinger (2019) have been particularly interesting in that they enabled a comparison between model behavior and human response, including both N400 effects and well-reasoned cloze judgments by human speakers. Despite being used in multiple studies since (Lialin et al, 2022;Rogers et al, 2020;Zhang et al, 2020), these datasets are quite small, ranging in size from 18 sentence pairs in negation (NEG-136-SIMP) to a maximum of 44 sentence pairs in the role-reversal dataset .…”

“…To understand how well language models perform on these extended datasets, we evaluated 22 models, including GPT3, following the methodology from Lialin et al (2022). Compared to the original test sets, we see a significant drop (up to 57%) in accuracy for both Role and Negation tasks.…”