Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations

Abstract

In attempts to "explain" predictions of machine learning models, researchers have proposed hundreds of techniques for attributing predictions to features that are deemed important. While these attributions are often claimed to hold the potential to improve human "understanding" of the models, surprisingly little work explicitly evaluates progress towards this aspiration. In this paper, we conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. They are challenged both to simulate the model on fresh reviews, and to edit reviews with the goal of lowering the probability of the originally predicted class. Successful manipulations would lead to an adversarial example. During the training (but not the test) phase, input spans are highlighted to communicate salience. Through our evaluation, we observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control. For the BERT-based classifier, popular local explanations do not improve their ability to reduce the model confidence over the no-explanation case. Remarkably, when the explanation for the BERT model is given by the (global) attributions of a linear model trained to imitate the BERT model, people can effectively manipulate the model. 1 * denotes equal contribution.

Human Interpretation of Saliency-based Explanation Over Text

“…The granularity provided in the scoring function may vary greatly, from a binary measure-important or not important-to a complete saliency map, depending on the tokenization granularity, the method and visualization. Most commonly, the explanation is given as a colorized saliency map over word tokens [e.g., 2,4,5,47,51]. Note that this work is not concerned with a particular feature-attribution method, but rather how feature-attribution explanations generally communicate information to human explainees, and what the explainees comprehend from them.…”

“…Feature-attribution explanations aim to convey which parts of the input to a model decision are "important", "responsible" or "influential" to the decision [3,7,36,42,60]. This class of explanation methods is a prevalent mode of describing NLP processes [10,31,36,47], due to two main strengths: (1) it is flexible and convenient, with many different measures developed which communicate some aspect of feature importance; (2) and it is intuitive, with-seemingly, as we discover-straightforward interfaces of relaying this information. Here we cover background on feature-attribution explanations on two fronts in alignment with these strengths: the underlying technologies (Section 2.1) and the information which they communicate to humans (Section 2.2).…”

“…What is the nature of any discrepancy in this perception? 2 As Miller [39] writes, literature in the social sciences about how humans comprehend explanations and behavior can help illuminate this problem.…”

“…In natural language processing (NLP), this refers to which words, phrases or sentences in the input contributed most to the model prediction [10,36]. While much research exists on developing and verifying such explanations [1,4,32,35,50,51],less is known about the information that human explainees actually understand from them [2,12,19,39].In the explainable NLP literature, it is generally (implicitly) assumed that the explainee interprets the information "correctly", as it is communicated [4,17,20]: e.g., when one word is explained to be influential in the model's decision process, or more influential than another word, it is assumed that the explainee understands this relationship [28].We question this assumption: research in the social sciences describes modes in which the human explainee may be biased-via some cognitive habit-in their interpretation of processes [15,37,39,52]. Additional research shows this effect manifests in practice in AI settings [11,14,22,25,40].…”

“…less is known about the information that human explainees actually understand from them [2,12,19,39].…”