Fooling Partial Dependence via Data Poisoning

Baniecki, Hubert; Kretowicz, Wojciech; Biecek, Przemysław

doi:10.48550/arxiv.2105.12837

Cited by 5 publications

(12 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• 𝐿 𝑝 Distance: An intuitive and straightforward metric to compare two explanation maps is to compute the normed distance between them. Some of the widely used metrics are median 𝐿 1 distance, used in [15], and the 𝐿 2 distance [17,36,107]. Mean Squared Error (MSE), a metric derived from 𝐿 2 distance, is also very popular.…”

Section: Evaluating the Robustness Of Explanation Methodsmentioning

confidence: 99%

“…(3) Trust attack (Figure 2c) leads to different predictions but similar explanations. (3) Tabular data (ML models) [17] -dataset is poisoned to conceal the suspected behavior (bias) . Deep neural network models are currently used as black boxes due to their high accuracy for various tasks.…”

Section: Attacking Explainability Methodsmentioning

confidence: 99%

“…to explain their predictions and an equal amount of effort to show how inherently "fragile" they are, along with defenses to fix it. After Ghorbani et al [45] showed that current gradient methods are vulnerable to an attack, others researched for approaches to attack other data types like text/NLP [55,108] (Figure 3) and tabular data [17]. Below we list a few standard techniques considered while attacking models, which have successfully proved the vulnerability [45] of current explanation methods.…”

Section: Common Techniques and Metricsmentioning

confidence: 99%

See 2 more Smart Citations

On the Robustness of Explanations of Deep Neural Network Models: A Survey

Jyoti¹,

Ganesh²,

Gayala³

et al. 2022

Preprint

View full text Add to dashboard Cite

Explainability has been widely stated as a cornerstone of the responsible and trustworthy use of machine learning models. With the ubiquitous use of Deep Neural Network (DNN) models expanding to risk-sensitive and safety-critical domains, many methods have been proposed to explain the decisions of these models. Recent years have also seen concerted efforts that have shown how such explanations can be distorted (attacked) by minor input perturbations. While there have been many surveys that review explainability methods themselves, there has been no effort hitherto to assimilate the different methods and metrics proposed to study the robustness of explanations of DNN models. In this work, we present a comprehensive survey of methods that study, understand, attack, and defend explanations of DNN models. We also present a detailed review of different metrics used to evaluate explanation methods, as well as describe attributional attack and defense methods. We conclude with lessons and take-aways for the community towards ensuring robust explanations of DNN model predictions.

show abstract

Section: Evaluating the Robustness Of Explanation Methodsmentioning

confidence: 99%

Section: Attacking Explainability Methodsmentioning

confidence: 99%

Section: Common Techniques and Metricsmentioning

confidence: 99%

See 1 more Smart Citation

On the Robustness of Explanations of Deep Neural Network Models: A Survey

Jyoti¹,

Ganesh²,

Gayala³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Let us consider the example of the Heart Disease dataset 1 , which deals with the binary classification for the presence of a heart disease. A Support Vector Classification model was built for this dataset and then a technique described by Baniecki et al (2021) was applied to perturb the dataset in a way that alters the PDP curve, which could be done maliciously to influence the result. The new perturbed dataset has the same dimension as the original dataset.…”

Section: Context-sensitive Explanationsmentioning

confidence: 99%

Do not explain without context: addressing the blind spot of model explanations

Woźnica,

Pękala,

Baniecki

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

The increasing number of regulations and expectations of predictive machine learning models, such as so called right to explanation, has led to a large number of methods promising greater interpretability. High demand has led to a widespread adoption of XAI techniques like Shapley values, Partial Dependence profiles or permutational variable importance. However, we still do not know enough about their properties and how they manifest in the context in which explanations are created by analysts, reviewed by auditors, and interpreted by various stakeholders. This paper highlights a blind spot which, although critical, is often overlooked when monitoring and auditing machine learning models: the effect of the reference data on the explanation calculation. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution.We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion. Consequently, we postulate that obtaining robust and useful explanations always requires supporting them with a broader context.

show abstract

“…Recently, several studies reported that such a manipulation is possible, e.g. by modifying the black-box model to be explained [8,9,10], by manipulating the computation algorithms of feature attributions [11,12], and by poisoning the data distribution [13]. With these findings in mind, the current possible advice to the auditors is not to rely solely on the reported feature attributes for fairness auditing.…”

Section: Introductionmentioning

confidence: 99%

Fooling SHAP with Stealthily Biased Sampling

Laberge¹,

Aïvodji²,

Hara³

et al. 2022

Preprint

View full text Add to dashboard Cite

SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus a background distribution. Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired explanations. However, existing attacks focus solely on altering the black-box model itself. In this paper, we propose a complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution. In the context of fairness audit, we show that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups, while remaining undetected. These results highlight the manipulability of SHAP explanations and encourage auditors to treat post-hoc explanations with skepticism.Preprint. Under review.

show abstract

Fooling Partial Dependence via Data Poisoning

Cited by 5 publications

References 29 publications

On the Robustness of Explanations of Deep Neural Network Models: A Survey

On the Robustness of Explanations of Deep Neural Network Models: A Survey

Do not explain without context: addressing the blind spot of model explanations

Fooling SHAP with Stealthily Biased Sampling

Contact Info

Product

Resources

About