The Explanation Game: Towards Prediction Explainability through Sparse Communication

Abstract

Explainability is a topic of growing importance in NLP. In this work, we provide a unified perspective of explainability as a communication problem between an explainer and a layperson about a classifier's decision. We use this framework to compare several explainers, including gradient methods, erasure, and attention mechanisms, in terms of their communication success. In addition, we reinterpret these methods in the light of classical feature selection, and use this as inspiration for new embedded explainers, through the use of selective, sparse attention. Experiments in text classification and natural language inference, using different configurations of explainers and laypeople (including both machines and humans), reveal an advantage of attention-based explainers over gradient and erasure methods, and show that selective attention is a simpler alternative to stochastic rationalizers. Human experiments show strong results on text classification with post-hoc explainers trained to optimize communication success.

SPECTRA: Sparse Structured Text Rationalization

“…For all rationalizers, we map each input word to 300D-pretrained GloVe embeddings from 840B release (Pennington et al, 2014) that are kept frozen. We instantiate all encoder networks as bidirectional LSTM (Hochreiter and Schmidhuber, 1997) layers (BiLSTM) (w/ hidden size 200) similarly to Lei et al (2016); Bastings et al (2019); Treviso and Martins (2020). Although other works (Jain et al, 2020;Paranjape et al, 2020) use more powerful BERT-based representations, we firstly experimented with BiLSTM layers and noticed our results were competitive with those reported in Jain et al (2020).…”

“…These approaches are often brittle and fragile for the high sensitivity that they show to changes in the hyperparameters and to variability due to sampling. On the other hand, existing rationalizers that use sparse attention mechanisms (Treviso and Martins, 2020) such as sparsemax attention, while being deterministic and end-to-end differentiable, do not have a direct handle to constrain the rationale in terms of sparsity and contiguity. We endow them with these capabilities in this paper as shown in Table 1, where we position our work in the literature for highlights extraction.…”

“…There is a long string of work on interpreting predictions made by neural networks (Lipton, 2017;Doshi-Velez and Kim, 2017;Gilpin et al, 2019;Wiegreffe and Marasović, 2021;Zhang et al, 2021a). Our paper focus on selective rationalizers, which have been used for extraction of text highlights (Lei et al, 2016;Bastings et al, 2019;Yu et al, 2019;DeYoung et al, 2019;Treviso and Martins, 2020;Zhang et al, 2021b) and text matchings (Swanson et al, 2020). Most works rely on stochastic rationale generation or deterministic attention mechanisms, but the two approaches have never been extensively compared.…”

“…Other works use strategies such as top-k to map token-level scores to rationales, but also require gradient estimations to train both modules jointly (Paranjape et al, 2020;Chang et al, 2020). In turn, sparse attention mechanisms (Treviso and Martins, 2020) are deterministic and have exact gradients, but lack a direct way to control sparsity and contiguity in the rationale extraction. This raises the question: how can we build an easy-to-train fully differentiable rationalizer that allows for flexible constrained rationale extraction?…”

Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification

“…Chen and Ji (2020) propose learning a variational word mask to improve model interpretability. Finally, extracting a short snippet from the original input text (rationale) and using it to make a prediction has been recently proposed (Lei et al, 2016;Bastings et al, 2019;Treviso and Martins, 2020;Jain et al, 2020;Chalkidis et al, 2021). Nguyen (2018) and Atanasova et al (2020) compare explanations produced by different approaches, showing that in most cases gradientbased approaches outperform sparse linear metamodels.…”