Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students?

Abstract

While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated. In this work, we introduce a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. Crucially, the explanations are available to the student during training, but are not available at test time. Compared with prior proposals, our approach is less easily gamed, enabling principled, automatic, model-agnostic evaluation of attributions. Using our framework, we compare numerous attribution methods for text classification and question answering, and observe quantitative differences that are consistent (to a moderate to high degree) across different student model architectures and learning strategies.1

A survey on improving NLP models with human explanations

“…Comparative studies We found almost no works that empirically compare approaches for learning from explanations across integration methods or explanation types. Pruthi et al (2022) compare MTL and REGULARIZATION methods for learning from HIGHLIGHT explanations. They find that the former method requires more training examples and slightly underperforms regularization.…”

“…Selecting informative explanations Based on experiments with an artificial dataset, Hase and Bansal (2021) conclude that a model can be improved based on explanations if it can infer relevant latent information better from input instance and explanation combined, than from the input instance alone. This property could be quantified according to the metric suggested by Pruthi et al (2022), who quantify explanation quality as the performance difference between a model trained on input instances and trained with additional explanation annotations. Carton et al (2021) find that models can profit from those highlight explanations which lead to accurate model predictions if presented to the model in isolation.…”

Learning to Scaffold: Optimizing Model Explanations for Teaching

“…In contrast, recent works have focused on more quantitative criteria: correlation between explainability methods for measuring consistency Wallace, 2019, Serrano andSmith, 2019], sufficiency and comprehensiveness [DeYoung et al, 2020], and simulability: whether a human or machine consumer of explanations understands the model behavior well enough to predict its output on unseen examples [Doshi-Velez and Kim, 2017]. Simulability, in particular, has a number of desirable properties, such as being intuitively aligned with the goal of communicating the underlying model behavior to humans and being measurable in manual and automated experiments [Treviso and Martins, 2020, Hase and Bansal, 2020, Pruthi et al, 2020. Figure 1: Illustration of our SMaT framework.…”

“…For instance, Pruthi et al [2020] proposed a framework for automatic evaluation of simulability that, given a teacher model and explanations of this model's predictions, trains a student model to match the teacher's predictions. The explanations are then evaluated with respect to how well they help a student learn to simulate the teacher ( § 2).…”

“…For instance, interpretability methods using saliency maps define E as the space of normalized distributions of importance over L input elements e ∈ L−1 (where L−1 is the (L − 1)-probability simplex). Pruthi et al [2020] proposed an automatic framework for evaluating explainers that trains a student model S θ : X → Y with parameters θ to simulate the teacher (i.e., the original classifier) in a constrained setting. For example, the student can be constrained to have less capacity than the teacher by using a simpler model or trained with a subset of the dataset used for the teacher ( Dtrain D train ).…”