Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

Abstract

Causal abstraction is a promising theoretical framework for explainable artificial intelligence that defines when an interpretable high-level causal model is a faithful simplification of a lowlevel deep learning system. However, existing causal abstraction methods have two major limitations: they require a brute-force search over alignments between the high-level model and the low-level one, and they presuppose that variables in the high-level model will align with disjoint sets of neurons in the low-level one. In this paper, we present distributed alignment search (DAS), which overcomes these limitations. In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in nonstandard bases-distributed representations. Our experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS removes previous obstacles to conducting causal abstraction analyses and allows us to find conceptual structure in trained neural nets.

Large language models as models of human cognition

“…The models are truly learning a symbol, rather than a weaker heuristic. 17 Another potential strategy is to examine the representations that emerge from particular learning inputs, and how these support behaviors. In humans, these learning inputs can only be manipulated in the context of short lab experiments, but with models the learning input can be changed radically in both its quantity and its contents.…”

Interpretability Is in the Mind of the Beholder: A Causal Framework for Human-Interpretable Representation Learning

“…(ii) a more general setting in which the human's concepts are constrained to be disentangled in blocks; and (iii) an unrestricted setting in which the human concepts can influence each other in arbitrary manners. In addition, we identify a and previously ignored link between interpretability of representations and the notion of causal abstraction [34][35][36].…”

“…The presence of a consistency property between C H and C M is what defines a causal abstraction [34,35,60]; see [61] for an overview. Causal abstractions have been proposed to define (approximate) equivalence between causal graphs and have recently been employed in the context of explainable AI [36,62]. The existence of a causal abstraction ensures two systems are interventionally equivariant: interventions on one system can always be mapped (modulo approximations) to equivalent interventions in the other and lead to the same interventional distribution.…”

“…In other words, alignment is a property of the map from H to Ĥ, while faithfulness is a property of the map between M and Ĥ. Evaluating faithfulness can be performed, e.g., via TCAV scores and assessing the degree of linear separation of concepts in the machine representation M. Another approach [36] measures the degree to which M can be reconducted to a causal decision graph on top of the concepts of interest. Other methods are discussed in Section 6.2.…”