Beyond Sparsity: Tree Regularization of Deep Models for Interpretability

Wu, Mike; Hughes, Michael C.; Parbhoo, Sonali; Zazzi, Maurizio; Röth, Volker; Doshi‐Velez, Finale

doi:10.48550/arxiv.1711.06178

Cited by 29 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, even a small neural model may not be interpretable. Closest to our work, [7] regularize a neural model to behave like a simple decision tree.…”

Section: Related Workmentioning

confidence: 99%

“…Instead of interpreting a model posthoc, an alternative is to optimize a measure of interpretability alongside predictive performance. [7], [11] pose two paths forward: include input gradient explanations or decision tree explanations in the objective function. As a result, models are encouraged to find "more interpretable" minima.…”

Section: Related Workmentioning

confidence: 99%

“…Works on global explanation (e.g. [7], [8]) return a single explanation for the entire model. Unfortunately, if the explanation is simple enough to be understandable, then it is unlikely to be faithful to the deep model across all inputs.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Regional Tree Regularization for Interpretability in Deep Neural Networks

Parbhoo

Hughes

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

The lack of interpretability remains a barrier to adopting deep neural networks across many safety-critical domains. Tree regularization was recently proposed to encourage a deep neural network's decisions to resemble those of a globally compact, axis-aligned decision tree. However, it is often unreasonable to expect a single tree to predict well across all possible inputs. In practice, doing so could lead to neither interpretable nor performant optima. To address this issue, we propose regional tree regularization – a method that encourages a deep model to be well-approximated by several separate decision trees specific to predefined regions of the input space. Across many datasets, including two healthcare applications, we show our approach delivers simpler explanations than other regularization schemes without compromising accuracy. Specifically, our regional regularizer finds many more “desirable” optima compared to global analogues.

show abstract

“…However, even a small neural model may not be interpretable. Closest to our work, [7] regularize a neural model to behave like a simple decision tree.…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Regional Tree Regularization for Interpretability in Deep Neural Networks

Parbhoo

Hughes

et al. 2020

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…The field of Explainable AI (XAI) has emerged specifically for research on the development of interpretable machine learning algorithms that can increase the transparency of black-box models [3,10,13,47,65]. While the majority of XAI techniques focus on expert systems designed for machine learning developers [50,51,62,64], a growing body of work at the intersection of XAI and HCI has developed explainable methodologies targeting non-technical users [11,16].…”

Section: Introductionmentioning

confidence: 99%

Explainable Activity Recognition for Smart Home Systems

Das¹,

Nishimura²,

Vivek³

et al. 2021

Preprint

View full text Add to dashboard Cite

Smart home environments are designed to provide services that help improve the quality of life for the occupant via a variety of sensors and actuators installed throughout the space. Many automated actions taken by a smart home are governed by the output of an underlying activity recognition system. However, activity recognition systems may not be perfectly accurate and therefore inconsistencies in smart home operations can lead a user to wonder "why did the smart home do that?" In this work, we build on insights from Explainable Artificial Intelligence (XAI) techniques to contribute computational methods for explainable activity recognition. Specifically, we generate explanations for smart home activity recognition systems that explain what about an activity led to the given classification. To do so, we introduce four computational techniques for generating natural language explanations of smart home data and compare their effectiveness at generating meaningful explanations. Through a study with everyday users, we evaluate user preferences towards the four explanation types. Our results show that the leading approach, SHAP, has a 92% success rate in generating accurate explanations. Moreover, 84% of sampled scenarios users preferred natural language explanations over a simple activity label, underscoring the need for explainable activity recognition systems. Finally, we show that explanations generated by some XAI methods can lead users to lose confidence in the accuracy of the underlying activity recognition model, while others lead users to gain confidence. Taking all studied factors into consideration, we make a recommendation regarding which existing XAI method leads to the best performance in the domain of smart home automation, and discuss a range of topics for future work in this area.

show abstract

“…Ismail et al (2020) demonstrate the unreliability and inaccuracy of several explanation methods designed for tabular data in identifying feature importance in temporal models. Some approaches have focused on interpreting specific temporal model architectures, such as recurrent neural networks (Karpathy et al, 2015;Suresh et al, 2017;Ismail et al, 2019) and attention-based models (Choi et al, 2016;Zhang et al, 2019), while others have explored methods to encourage temporal models during training to be more interpretable using tree regularization (Wu et al, 2017) and game-theoretic characterizations (Lee et al, 2018). However, model-agnostic explanation for temporal models has begun to be addressed only recently.…”

Section: Introductionmentioning

confidence: 99%

Feature Importance Explanations for Temporal Black-Box Models

Sood¹,

Craven²

2021

Preprint

View full text Add to dashboard Cite

Models in the supervised learning framework may capture rich and complex representations over the features that are hard for humans to interpret. Existing methods to explain such models are often specific to architectures and data where the features do not have a time-varying component. In this work, we propose TIME, a method to explain models that are inherently temporal in nature. Our approach (i) uses a model-agnostic permutationbased approach to analyze global feature importance, (ii) identifies the importance of salient features with respect to their temporal ordering as well as localized windows of influence, and (iii) uses hypothesis testing to provide statistical rigor.

show abstract

Beyond Sparsity: Tree Regularization of Deep Models for Interpretability

Cited by 29 publications

References 0 publications

Regional Tree Regularization for Interpretability in Deep Neural Networks

Regional Tree Regularization for Interpretability in Deep Neural Networks

Explainable Activity Recognition for Smart Home Systems

Feature Importance Explanations for Temporal Black-Box Models

Contact Info

Product

Resources

About