Explaining by Removing: A Unified Framework for Model Explanation

Abstract

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We establish a new class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 25 existing methods, including several of the most widely used approaches (SHAP, LIME, Meaningful Perturbations, permutation tests). This new class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.

FastSHAP: Real-Time Shapley Value Estimation

“…where the held out features x 1−s are marginalized out using their joint marginal distribution p(x 1−s ) and a link function (e.g., logit) is applied to the model output. Recent work has debated the properties of different value function formulations, particularly the choice of how to remove features [1,19,9,14]. Regardless of the formulation, this approach to model explanation enjoys several useful theoretical properties due to the use of Shapley values: for example, they are zero for irrelevant features and are guaranteed to sum to the model's prediction.…”

“…First, many works have proposed stochastic estimators [3,40,39,27,10,44] that rely on sampling either feature subsets or permutations; these are often consistent estimators, but they require many model evaluations and involve a trade-off between run-time and accuracy. Second, some works have proposed model-specific approximations, e.g., for trees [26] or neural networks [35,6,2,43]; these are generally faster, but they sometimes require many model evaluations, often induce bias, and typically lack flexibility regarding how to handle held-out features when generating explanations-a subject of continued debate in the field [1,19,9,14].…”

Fast Hierarchical Games for Image Explanations

“…4. We remark that one of the most important details of any explanation method based on feature removal is the baseline, which defines the value that X C takes in the entries not in C. There are different approaches to removing features, ranging from using the default value of 0, to using their conditional distribution (refer to [9] for further details). Computing the latter can be challenging, and recent work has explored various approximations [1,11].…”

“…as baseline), h-Shap uses their expected value (or unconditional distribution [20]) for simplicity, as done by other works [9]. As pointed out by [9,27], this is valid under the assumptions of model linearity and feature independence 3 . Yet, as we will argue later in Sec.…”

“…To our knowledge, Covert et al[9] compiled the most comprehensive review of currently available explanation methods based on feature removal.…”