Learning to Deceive with Attention-Based Explanations

Abstract

Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention's reliability as a tool for auditing algorithms in the context of fairness and accountability. 1

Gradient-based Analysis of NLP Models is Manipulable

“…for all examples in the validation set. This metric estimates the extent to which the model is attributing its predictions to gender (an unbiased model should have less of this attribution), and is similar to the measure of bias used by Pruthi et al (2020).…”

“…We instead manipulate explanation methods to evaluate the extent to which a model's true reasoning can be hidden. Pruthi et al (2020) manipulate attention distributions in an end-to-end fashion; we focus on manipulating gradients. It is worth noting that we perturb models to manipulate interpretations; other work perturbs inputs (Ghorbani et al, 2019;Dombrowski et al, 2019;Subramanya et al, 2019).…”

“…We follow the setup of Pruthi et al (2020) and only use examples with the labels of "physician" and "surgeon". We also subsample female surgeons and male physicians by a factor of 10.…”

“…model. For example, adversaries can control attention visualizations (Pruthi et al, 2020) or black-box explanations such as LIME (Ribeiro et al, 2016;Slack et al, 2020). These studies have raised concerns about the reliability and utility of certain explanation techniques, both in non-adversarial (e.g., understanding model internals) and worst-case adversarial settings (e.g., concealing model biases from regulatory agencies).…”

Less is More: Attention Supervision with Counterfactuals for Text Classification

“…As for human consuming attention as explanation, there has been criticism that unsupervised attention weights are too poorly correlated with the contribution of each word for machine decision (or, unfaithful) (Jain and Wallace, 2019;Serrano and Smith, 2019;Pruthi et al, 2019). Meanwhile, (Wiegreffe and Pinter, 2019) develops diagnostics to decide when attention is good enough as explanation.…”