Sorting through the noise: Testing robustness of information processing in pre-trained language models

Abstract

Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this paper we tackle a component of this question by examining robustness of models' ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models' use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.

Large Language Models Demonstrate the Potential of Statistical Learning in Language

“…Through careful analyses of their output, we can assess just how much can be learned from the statistical regularities of the linguistic environment (Futrell et al., 2019; Wilcox, Futrell, & Levy, 2022). Some of this work has already been done in the context of encoder‐only masked language models, such as BERT and its related descendants (Ettinger, 2020; Pandia & Ettinger, 2021; Rogers, Kovaleva, & Rumshisky, 2020). Their failures, such as with semantic coherence or pragmatics (Arehalli, Dillon, & Linzen, 2022; Dou, Forbes, Koncel‐Kedziorski, Smith, & Choi, 2022; McClelland et al., 2020), are also interesting and point to other central tenets of usage‐based theories such as the role of environmental contexts, developmental histories, cognitive machinery, and functional pressures in human language learning and use (Christiansen & Chater, 2022).…”

COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models

“…Through our experiments, we found this criterion to be met in a majority of the cases, suggesting a strong capacity of models to demonstrate property inheritance. However, post-hoc analyses revealed that for most models, this capacity drastically decreases in the presence of distracting information (sometimes even worse than random-guessing), suggesting a clear lack of robustness in the information processing capacities of PLMs, similar to the results of Pandia and Ettinger (2021). In contrast to their results, we find that larger models are generally more distracted than are smaller models, and this especially happens when the distracting information is closer to the predicted property-phrases, suggesting the presence of a proximity effect.…”

“…To what extent does the compatibility of models with H1 hold in presence of distracting information? This question is inspired by Pandia and Ettinger (2021), who report a substantial decrease in perceived information processing capacity of PLMs in the presence of semantic distractors. Here, we transform the stimuli of COMPS-WUGS by creating two different subordinates for every minimal pair: one for the positive concept (e.g.…”

Computational Language Modeling and the Promise of In Silico Experimentation

“…This again suggests some isomorphism between human language processing and DL-based models. The next word prediction objective also enables language models to perform well on psycholinguistic diagnostics like the cloze task, although there is substantial room for improvement ( Ettinger, 2020 ; Pandia & Ettinger, 2021 ). Finally, self-supervised ANNs, that is, networks that predict the next word or speech frame, transfer well to downstream language tasks like question answering and coreference resolution, and to speech tasks like speaker verification and translation across languages ( Z. Chen et al, 2022 ; A. Wu et al, 2020 ).…”