L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data

Chen, Jianbo; Song, Lixin; Wainwright, Martin J.; Jordan, Michael I.

doi:10.48550/arxiv.1808.02610

Cited by 31 publications

(43 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, we carry out the mask-k-pixels experiments (Chen et al, 2018b) to demonstrate the equivariance of SITE as a self-interpretable model. This experiment is implemented by masking k pixels of the input data based on the interpretations provided.…”

Section: Comparison With Post-hoc Methodsmentioning

confidence: 99%

Self-Interpretable Model with TransformationEquivariant Interpretation

Wang¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

With the proliferation of machine learning applications in the real world, the demand for explaining machine learning predictions continues to grow especially in high-stakes fields. Recent studies have found that interpretation methods can be sensitive and unreliable, where the interpretations can be disturbed by perturbations or transformations of input data. To address this issue, we propose to learn robust interpretations through transformation equivariant regularization in a self-interpretable model. The resulting model is capable of capturing valid interpretations that are equivariant to geometric transformations. Moreover, since our model is self-interpretable, it enables faithful interpretations that reflect the true predictive mechanism. Unlike existing self-interpretable models, which usually sacrifice expressive power for the sake of interpretation quality, our model preserves the high expressive capability comparable to the state-of-the-art deep learning models in complex tasks, while providing visualizable and faithful highquality interpretation. We compare with various related methods and validate the interpretation quality and consistency of our model.

show abstract

Section: Comparison With Post-hoc Methodsmentioning

confidence: 99%

Self-Interpretable Model with TransformationEquivariant Interpretation

Wang¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…aggregation based methods [28] and Monte Carlo sampling [29]. There are also approaches for graph-structured data such as natural language text and images [30].…”

Section: Feature Importance Methodsmentioning

confidence: 99%

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Jia¹,

McDermid²,

Lawton³

et al. 2021

Preprint

View full text Add to dashboard Cite

Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing machine learning (ML). In many cases, ML is used on ill-defined problems, e.g. optimising sepsis treatment, where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the "opaque" nature of ML where the learnt model is not amenable to human scrutiny. Explainable AI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, there is not much work explicitly investigating the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which explainable AI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how explainable AI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, explainable AI methods can contribute to a safety case. Overall, we conclude that explainable AI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.

show abstract

“…Selective rationalization: [27] proposes the first generator-predictor framework for rationalization. Following this work, new game-theoretic frameworks were proposed to encourage different desired properties of the selected rationales, such as optimized Shapley structure scores [14], comprehensiveness [46], multi-aspect supports [4,11] and invariance [12]. Another fundamental direction is to overcome the training difficulties.…”

Section: Related Workmentioning

confidence: 99%

“…Selective rationalization [8,10,11,13,14,17,27,29,46] explains the prediction of complex neural networks by finding a small subset of the input -rationale -that suffices on its own to yield the same outcome as to the original data. To generate high-quality rationales, existing methods often train a cascaded system that consists of two components, i.e., a rationale generator and a predictor.…”

Section: Introductionmentioning

confidence: 99%

Understanding Interlocking Dynamics of Cooperative Rationalization

Yu¹,

Zhang²,

Chang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Selective rationalization explains the prediction of complex neural networks by finding a small subset of the input that is sufficient to predict the neural model output. The selection mechanism is commonly integrated into the model itself by specifying a two-component cascaded system consisting of a rationale generator, which makes a binary selection of the input features (which is the rationale), and a predictor, which predicts the output based only on the selected features. The components are trained jointly to optimize prediction performance. In this paper, we reveal a major problem with such cooperative rationalization paradigm -model interlocking. Interlocking arises when the predictor overfits to the features selected by the generator thus reinforcing the generator's selection even if the selected rationales are sub-optimal. The fundamental cause of the interlocking problem is that the rationalization objective to be minimized is concave with respect to the generator's selection policy. We propose a new rationalization framework, called A2R, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The generator now realizes both soft and hard attention over the features and these are fed into the two different predictors. While the generator still seeks to support the original predictor performance, it also minimizes a gap between the two predictors. As we will show theoretically, since the attention-based predictor exhibits a better convexity property, A2R can overcome the concavity barrier. Our experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem and find explanations that better align with human judgments. 2 * Authors contributed equally to this paper. Work was done when SC was at MIT-IBM Watson AI Lab. 2 We release our code at https://github.com/Gorov/Understanding_Interlocking.Preprint. Under review.

show abstract

L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data

Cited by 31 publications

References 7 publications

Self-Interpretable Model with TransformationEquivariant Interpretation

Self-Interpretable Model with TransformationEquivariant Interpretation

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Understanding Interlocking Dynamics of Cooperative Rationalization

Contact Info

Product

Resources

About