Finding and removing Clever Hans: Using explanation methods to debug and improve deep models

Anders, Christopher J.; Neumann, David; Samek, Wojciech; Müller, Klaus‐Robert; Lapuschkin, Sebastian

doi:10.1016/j.inffus.2021.07.015

Cited by 76 publications

(58 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the previous sections, we achieved this by averaging relevance patterns across all samples of a given class. To perform a more detailed analysis that is able to identify different learning strategies of the ML models, we propose the use of SpRAy [35] as described in [5] for clinical gait data.…”

Section: Do Different ML Methods Rely On Different Patterns?mentioning

confidence: 99%

“…Sets of samples with similar relevance patterns are tightly grouped together in the spectral embedding space, while samples with dissimilar patterns are located far apart. Together with the suggested cluster labels, the analytically derived solution in Φ can then be visualized in R 2 , e.g., via a t-SNE projection [5,39]. We implemented and evaluated SpRAy using the CoRelAy 3 framework [4] for Python.…”

Section: Do Different ML Methods Rely On Different Patterns?mentioning

confidence: 99%

See 1 more Smart Citation

Explaining Machine Learning Models for Clinical Gait Analysis

Slijepčević

Horst

Lapuschkin

et al. 2021

ACM Trans. Comput. Healthcare

Self Cite

View full text Add to dashboard Cite

Machine Learning (ML) is increasingly used to support decision-making in the healthcare sector. While ML approaches provide promising results with regard to their classification performance, most share a central limitation, their black-box character. This article investigates the usefulness of Explainable Artificial Intelligence (XAI) methods to increase transparency in automated clinical gait classification based on time series. For this purpose, predictions of state-of-the-art classification methods are explained with a XAI method called Layer-wise Relevance Propagation (LRP). Our main contribution is an approach that explains class-specific characteristics learned by ML models that are trained for gait classification. We investigate several gait classification tasks and employ different classification methods, i.e., Convolutional Neural Network, Support Vector Machine, and Multi-layer Perceptron. We propose to evaluate the obtained explanations with two complementary approaches: a statistical analysis of the underlying data using Statistical Parametric Mapping and a qualitative evaluation by two clinical experts. A gait dataset comprising ground reaction force measurements from 132 patients with different lower-body gait disorders and 62 healthy controls is utilized. Our experiments show that explanations obtained by LRP exhibit promising statistical properties concerning inter-class discriminativity and are also in line with clinically relevant biomechanical gait characteristics.

show abstract

Section: Do Different ML Methods Rely On Different Patterns?mentioning

confidence: 99%

Section: Do Different ML Methods Rely On Different Patterns?mentioning

confidence: 99%

Explaining Machine Learning Models for Clinical Gait Analysis

Slijepčević

Horst

Lapuschkin

et al. 2021

ACM Trans. Comput. Healthcare

Self Cite

View full text Add to dashboard Cite

show abstract

“…To overcome this limitation, [21,25,26] proposed the layer-wise relevance propagation and [45] proposed GradCAM as alternative strategies to visualize the reasons for individual CNN predictions. These methods can be extended for explaining multiple decisions and were used for investigating the clever-Hans phenomenon [27,46].…”

Section: Related Workmentioning

confidence: 99%

“…Eigenfaces [41] PCA Model parameter investigation LRP [21,25,26], SpRAy [27,46] Deep CNNs Post hoc CNN prediction analysis…”

Section: Models Explanation Strategymentioning

confidence: 99%

“…In order to explain the decision of deep learning models, the machine learning community proposed strategies [23][24][25] as well as methods such as the layer-wise relevance propagation [25,26]. Those methods mainly explain decisions of CNNs or gather these decisions for large databases [27]. Nevertheless, the explanatory factors coded in the models remains invisible in the models.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Investigating Explanatory Factors of Machine Learning Models for Plant Classification

Wöber

Mehnen

Sykacek

et al. 2021

Plants

View full text Add to dashboard Cite

Recent progress in machine learning and deep learning has enabled the implementation of plant and crop detection using systematic inspection of the leaf shapes and other morphological characters for identification systems for precision farming. However, the models used for this approach tend to become black-box models, in the sense that it is difficult to trace characters that are the base for the classification. The interpretability is therefore limited and the explanatory factors may not be based on reasonable visible characters. We investigate the explanatory factors of recent machine learning and deep learning models for plant classification tasks. Based on a Daucus carota and a Beta vulgaris image data set, we implement plant classification models and compare those models by their predictive performance as well as explainability. For comparison we implemented a feed forward convolutional neuronal network as a default model. To evaluate the performance, we trained an unsupervised Bayesian Gaussian process latent variable model as well as a convolutional autoencoder for feature extraction and rely on a support vector machine for classification. The explanatory factors of all models were extracted and analyzed. The experiments show, that feed forward convolutional neuronal networks (98.24% and 96.10% mean accuracy) outperforms the Bayesian Gaussian process latent variable pipeline (92.08% and 94.31% mean accuracy) as well as the convolutional autoenceoder pipeline (92.38% and 93.28% mean accuracy) based approaches in terms of classification accuracy, even though not significant for Beta vulgaris images. Additionally, we found that the neuronal network used biological uninterpretable image regions for the plant classification task. In contrast to that, the unsupervised learning models rely on explainable visual characters. We conclude that supervised convolutional neuronal networks must be used carefully to ensure biological interpretability. We recommend unsupervised machine learning, careful feature investigation, and statistical feature analysis for biological applications.

show abstract

Adversarial Attacks in Explainable Machine Learning: A Survey of Threats Against Models and Humans

Vadillo,

Santana,

Lozano

2024

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Reliable deployment of machine learning models such as neural networks continues to be challenging due to several limitations. Some of the main shortcomings are the lack of interpretability and the lack of robustness against adversarial examples or out‐of‐distribution inputs. In this paper, we comprehensively review the possibilities and limits of adversarial attacks for explainable machine learning models. First, we extend the notion of adversarial examples to fit in explainable machine learning scenarios where a human assesses not only the input and the output classification, but also the explanation of the model's decision. Next, we propose a comprehensive framework to study whether (and how) adversarial examples can be generated for explainable models under human assessment. Based on this framework, we provide a structured review of the diverse attack paradigms existing in this domain, identify current gaps and future research directions, and illustrate the main attack paradigms discussed. Furthermore, our framework considers a wide range of relevant yet often ignored factors such as the type of problem, the user expertise or the objective of the explanations, in order to identify the attack strategies that should be adopted in each scenario to successfully deceive the model (and the human). The intention of these contributions is to serve as a basis for a more rigorous and realistic study of adversarial examples in the field of explainable machine learning.

show abstract

Finding and removing Clever Hans: Using explanation methods to debug and improve deep models

Cited by 76 publications

References 48 publications

Explaining Machine Learning Models for Clinical Gait Analysis

Explaining Machine Learning Models for Clinical Gait Analysis

Investigating Explanatory Factors of Machine Learning Models for Plant Classification

Adversarial Attacks in Explainable Machine Learning: A Survey of Threats Against Models and Humans

Contact Info

Product

Resources

About