Interpreting Neural Networks Using Flip Points

Yousefzadeh, Roozbeh; O’Leary, Dianne P.

doi:10.48550/arxiv.1903.08789

Cited by 4 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moving the decision boundaries away from the training data also tends to improve the generalization of deep learning models as reported by Elsayed et al [3] and Yousefzadeh and O'Leary [35].…”

Section: Debugging a Modelmentioning

confidence: 84%

“…Here, we demonstrate our techniques for explaining, auditing, and debugging deep learning models on three different datasets with societal themes. We use three software packages, NLopt [10], IPOPT [30], and the Optimization Toolbox of MATLAB, as well as our own custom-designed homotopy algorithm [35], to solve the optimization problems. The algorithms almost always converge to the same point.…”

Section: Resultsmentioning

confidence: 99%

“…For future work, we would consider models with continuous outputs other than classification models, for example, a model that recommends the dose of a drug for patients. Other directions of research include auditing image classification models, expanding on work in [35], and text analysis models that have a societal impact. Our methods can promote fairness, accountability and transparency in deep learning models.…”

Section: Discussionmentioning

confidence: 99%

“…Extensions to models with multi-class outputs or quantified output is straightforward. We first review the work on flip points in [35] and then define constrained flip points.…”

Section: Defining and Computing Flip Pointsmentioning

confidence: 99%

See 3 more Smart Citations

Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis

Yousefzadeh,

O'Leary

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Deep learning models have been criticized for their lack of easy interpretation, which undermines confidence in their use for important applications. Nevertheless, they are consistently utilized in many applications, consequential to humans' lives, mostly because of their better performance. Therefore, there is a great need for computational methods that can explain, audit, and debug such models. Here, we use flip points to accomplish these goals for deep learning models with continuous output scores (e.g., computed by softmax), used in social applications. A flip point is any point that lies on the boundary between two output classes: e.g. for a model with a binary yes/no output, a flip point is any input that generates equal scores for "yes" and "no". The flip point closest to a given input is of particular importance because it reveals the least changes in the input that would change a model's classification, and we show that it is the solution to a well-posed optimization problem. Flip points also enable us to systematically study the decision boundaries of a deep learning classifier. The resulting insight into the decision boundaries of a deep model can clearly explain the model's output on the individual-level, via an explanation report that is understandable by non-experts. We also develop a procedure to understand and audit model behavior towards groups of people. Flip points can also be used to alter the decision boundaries in order to improve undesirable behaviors. We demonstrate our methods by investigating several models trained on standard datasets used in social applications of machine learning. We also identify the features that are most responsible for particular classifications and misclassifications. CCS CONCEPTS• Computing methodologies → Machine learning; • Humancentered computing → Social recommendation.

show abstract

Section: Debugging a Modelmentioning

confidence: 84%

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

“…Extensions to models with multi-class outputs or quantified output is straightforward. We first review the work on flip points in [35] and then define constrained flip points.…”

Section: Defining and Computing Flip Pointsmentioning

confidence: 99%

See 2 more Smart Citations

Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis

Yousefzadeh,

O'Leary

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, the two methods require a lot of pixels to be changed. Yousefzadeh and O'Leary Yousefzadeh and O'Leary [2019] reduced the number of pixels using flip points. It is al possible to deceive a neural network classifier with only one pixel change Su et al [2019].…”

Section: Adversarial Examplesmentioning

confidence: 99%

Demystifying Deep Neural Networks Through Interpretation: A Survey

Dao,

Lee

2020

Preprint

View full text Add to dashboard Cite

Modern deep learning algorithms tend to optimize an objective metric, such as minimize a cross entropy loss on a training dataset, to be able to learn. The problem is that the single metric is an incomplete description of the real world tasks. The single metric cannot explain why the algorithm learn. When an erroneous happens, the lack of interpretability causes a hardness of understanding and fixing the error. Recently, there are works done to tackle the problem of interpretability to provide insights into neural networks behavior and thought process. The works are important to identify potential bias and to ensure algorithm fairness as well as expected performance.

show abstract

Explainable machine learning in cybersecurity: A survey

Yan

Wen

Nepal

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Machine learning (ML) techniques are increasingly important in cybersecurity, as they can quickly analyse and identify different types of threats from millions of events. In spite of the increasing number of possible applications of ML, successful adoption of ML models in cybersecurity still highly relies on the explainability of those models that are used for making predictions. Explanations that support ML model outputs are crucial in cybersecurity‐oriented ML applications because people need to get more information from the model than just binary output for analysis. The explainable models help ML developers solve the “trust” problem for a security application prediction in a faithful way: validating model behaviours, diagnosing misclassifications and sometimes automatically patching errors in the target models. Therefore, explainable ML for cybersecurity has become a necessary and important research branch. In this paper, we present the topic of explainable ML in cybersecurity through two general types of explanations: (1) ante hoc explanation, and (2) post hoc explanation, with their methodologies. We systematically review and categorise the state‐of‐the‐art research, and provide comparative studies to help researchers find the optimal solutions to specific problems. We further list open issues in this field to facilitate future studies. This survey will benefit diverse groups of readers from both academia and industries, who want to effectively use ML to solve cybersecurity challenges.

show abstract

Interpreting Neural Networks Using Flip Points

Cited by 4 publications

References 0 publications

Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis

Auditing and Debugging Deep Learning Models via Decision Boundaries: Individual-level and Group-level Analysis

Demystifying Deep Neural Networks Through Interpretation: A Survey

Explainable machine learning in cybersecurity: A survey

Contact Info

Product

Resources

About