What Context Features Can Transformer Language Models Use?

Abstract

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid-and longrange contexts, we find that several extremely destructive context manipulations-including shuffling word order within sentences and deleting all words other than nouns-remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models. 1

Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Responses in the Language Network

“…Specifically, we examined versions made up of: (i) all the content words, that is, nouns, verbs, adjectives, and adverbs ( KeepContentW ); (ii) nouns, verbs, and adjectives ( KeepNVA ); (iii) nouns and verbs ( KeepNV ); (iv) nouns ( KeepN ); and (v) only the function words ( KeepFunctionW ). Following O’Connor and Andreas (2021) , we included pronouns and proper names in the set of nouns. Note also that because not all the sentences had adverbs and/or adjectives, some pairs of the conditions (i), (ii), (iii), and (iv) could be identical for some sentences.…”

“…The perturbation manipulation conditions that we use in the current work are motivated by prior theorizing in language research and/or past empirical findings from both neuroscience and natural language processing (NLP). The perturbation manipulations include (i) word-order manipulations of varying severity that preserve or destroy local dependency structure (following Mollica et al, 2020 ), allowing us to investigate the effect of word order degradation while controlling for local word co-occurrence statistics; (ii) information-loss manipulations with deletion of words of different parts of speech (following O’Connor & Andreas, 2021 ), allowing us to investigate loss of information from particular classes of words; (iii) semantic-distance manipulations with sentence substitutions that relate to the meaning of the original sentence to varying degrees (inspired by Pereira et al, 2018 ), allowing us to investigate loss of semantic and more general topical information while retaining sentence well-formedness. As a baseline length-matched control condition, we include a random word list, where each word is substituted with a different random word.…”

“…Indeed, previous co-occurrence-based models often benefited from the removal of function words and other high-frequency “stop words” for solving various NLP tasks (e.g., Bernardi et al, 2013 ; Herbelot & Baroni, 2017 ; Lazaridou et al, 2017 ). Even for state-of-the-art ANN language models, it has recently been shown that retaining only the content words in linguistic context has little effect on next-word prediction performance, with performance varying as a function of how much of the lexical content is included (i.e., higher performance when keeping all content words vs. keeping only subsets, such as keeping only the nouns and verbs, or keeping only the nouns; O’Connor & Andreas, 2021 ). Whereas function words have been shown to have a sizable effect on ANN next-word prediction performance within local sentence contexts (because they help ensure grammaticality; Khandelwal et al, 2018 ), content words strongly influence prediction performance both within local and more extended contexts ( Khandelwal et al, 2018 ; O’Connor & Andreas, 2021 ).…”

“…In line with this previous research, we hypothesized that removing content words, but not function words, should have a strong negative effect on ANN-to-brain mapping and that brain predictivity should increase the more lexical content of the original sentence is available to the ANN for building a representation. Following O’Connor and Andreas (2021) , we evaluate the effect of preserving all or some of the content words (e.g., nouns and verbs) or function words.…”

A Bayesian Framework for Information-Theoretic Probing

“…This distinction has been explicitly noted before, by e.g Bartlett (1953)(Pimentel et al, 2019(Pimentel et al, , 2020bBugliarello et al, 2020;McAllester and Stratos, 2020;Torroba Hennigen et al, 2020;Fernandes et al, 2021;O'Connor and Andreas, 2021). In those works, though, it was usually interpreted as a computational approximation to the truth-MI (or to V-information(Xu et al, 2020), which is discussed later in the paper).…”