Implicit Representations of Meaning in Neural Language Models

Li, Belinda Z.; Nye, Maxwell; Andreas, Jacob

doi:10.18653/v1/2021.acl-long.143

Cited by 37 publications

(31 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The tasks in this paper can be viewed as exploring one criticism of large language models, namely, to what extent do they simply rely on surface-level statistical correlations on text, without learning semantics or world knowledge (Bender & Koller, 2020)? In response, Li et al (2021) provide evidence that pre-trained language models do indeed construct approximate representations of the semantics of the situations they describe in text. In the context of programs, Austin et al (2021) approach this question by exploring the learning to execute task on MBPP, which we consider in Section 5.2.…”

Section: Related Workmentioning

confidence: 92%

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Nye¹,

Andreassen²,

Gur-Ari³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text (Brown et al., 2020) or synthesizing computer programs Austin et al., 2021). However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021). Surprisingly, we find that these same models are able to perform complex multistep computations-even in the few-shot regime-when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train Transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

show abstract

Section: Related Workmentioning

confidence: 92%

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Nye¹,

Andreassen²,

Gur-Ari³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Geiger et al (2020) initially used interchange interventions to show that a BERT-based natural language inference model creates a neural representation of lexical entailment, arguing that the ability to create modular representations underlies the capacity for systematic generalization. Li et al (2021) use interchange interventions to show that neural representations represent propositional content that systematically alters text generated by a neural language model. Geiger et al (2021) explicitly ground interchange intervention analysis in a theory of causal abstraction.…”

Section: Methods For Explaining the Internal Structure Of Aimentioning

confidence: 99%

Causal Abstraction for Faithful Model Interpretation

Geiger¹,

Potts²,

Icard³

2023

Preprint

View full text Add to dashboard Cite

A faithful and interpretable explanation of an AI model's behavior and internal structure is a high-level explanation that is human-intelligible but also consistent with the known, but often opaque low-level causal details of the model. We argue that the theory of causal abstraction provides the mathematical foundations for the desired kinds of model explanations. In causal abstraction analysis, we use interventions on model-internal states to rigorously assess whether an interpretable high-level causal model is a faithful description of an AI model. Our contributions in this area are: (1) We generalize causal abstraction to cyclic causal structures and typed high-level variables. (2) We show how multi-source interchange interventions can be used to conduct causal abstraction analyses. (3) We define a notion of approximate causal abstraction that allows us to assess the degree to which a high-level causal model is a causal abstraction of a lower-level one. (4) We prove constructive causal abstraction can be decomposed into three operations we refer to as marginalization, variable-merge, and value-merge. (5) We formalize the XAI methods of LIME, causal effect estimation, causal mediation analysis, iterated nullspace projection, and circuit-based explanations as special cases of causal abstraction analysis.

show abstract

“…Comparison to probing results Recently, Li et al (2021) developed a probe for investigating whether LM representations provide information about the state of entities at various stages in a larger discourse. This probing method-like the ones presented in this work-also aims to assess entity tracking abilities of pre-trained language mod-els.…”

Section: Nlg Evaluationmentioning

confidence: 99%

“…First, the probing classifier was trained on data that was similar to the evaluation data and this setup therefore provided a lot of supervision. Second, the datasets used by Li et al (2021) were obtained through crowdsourcing or a generation engine and were not constructed as systematically as ours. For these reasons, the probing classifier may have learned spurious correlations between the training and test splits, and the high accuracy on the task may have only in part been driven by entity tracking abilities of LMs.…”

Section: Nlg Evaluationmentioning

confidence: 99%

When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it

Schuster¹,

Linzen²

2022

Preprint

View full text Add to dashboard Cite

Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases (NPs), such as a dog, frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, a dog in Arthur doesn't own a dog does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite NPs. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple NPs and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.

show abstract

Implicit Representations of Meaning in Neural Language Models

Cited by 37 publications

References 22 publications

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Causal Abstraction for Faithful Model Interpretation

When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to it

Contact Info

Product

Resources

About