Evaluating Explanation Methods for Deep Learning in Security

Warnecke, Alexander; Arp, Daniel J.; Wressnegger, Christian; Rieck, Konrad

doi:10.1109/eurosp48549.2020.00018

Cited by 78 publications

(127 citation statements)

References 23 publications

Supporting

Mentioning

125

Contrasting

Order By: Relevance

“…An essential precondition is having fine-grained analysis results, e.g., highlighting problematic code tokens (Russell et al ,[S2]), rather than declaring an entire code sample vulnerable. For example, layer relevance propagation, an explanation technique propagating the prediction of neural networks layer-wise back to its inputs, could be utilized to report which tokens influenced the current decision of a model (Warnecke et al 2020). Such methods would allow highlighting the most problematic code locations to a user and to guide further inspection and should be explored for vulnerability analysis (Zou et al (2019) [S6], [S12]).…”

Section: Explainability Of Analysis Resultsmentioning

confidence: 99%

Deep security analysis of program code

2021

View full text Add to dashboard Cite

Due to the continuous digitalization of our society, distributed and web-based applications become omnipresent and making them more secure gains paramount relevance. Deep learning (DL) and its representation learning approach are increasingly been proposed for program code analysis potentially providing a powerful means in making software systems less vulnerable. This systematic literature review (SLR) is aiming for a thorough analysis and comparison of 32 primary studies on DL-based vulnerability analysis of program code. We found a rich variety of proposed analysis approaches, code embeddings and network topologies. We discuss these techniques and alternatives in detail. By compiling commonalities and differences in the approaches, we identify the current state of research in this area and discuss future directions. We also provide an overview of publicly available datasets in order to foster a stronger benchmarking of approaches. This SLR provides an overview and starting point for researchers interested in deep vulnerability analysis on program code.

show abstract

Section: Explainability Of Analysis Resultsmentioning

confidence: 99%

Deep security analysis of program code

2021

View full text Add to dashboard Cite

show abstract

“…Fan et al [23] assess the quality of five interpretation techniques in Android malware analysis applications, where they evaluate the stability, robustness, and effectiveness of the interpretations. Warnecke et al [81] also study the similar dimensions of interpretation in security domain. They both find that different interpretation techniques could generate different interpretation results for the same prediction result.…”

Section: Reproducibility Of Machine Learningmentioning

confidence: 94%

Towards a Consistent Interpretation of AIOps Models

LyuYingzhe

Rajbahadur

Lin

et al. 2021

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Artificial Intelligence for IT Operations (AIOps) has been adopted in organizations in various tasks, including interpreting models to identify indicators of service failures. To avoid misleading practitioners, AIOps model interpretations should be consistent (i.e., different AIOps models on the same task agree with one another on feature importance). However, many AIOps studies violate established practices in the machine learning community when deriving interpretations, such as interpreting models with suboptimal performance, though the impact of such violations on the interpretation consistency has not been studied. In this article, we investigate the consistency of AIOps model interpretation along three dimensions: internal consistency, external consistency, and time consistency. We conduct a case study on two AIOps tasks: predicting Google cluster job failures and Backblaze hard drive failures. We find that the randomness from learners, hyperparameter tuning, and data sampling should be controlled to generate consistent interpretations. AIOps models with AUCs greater than 0.75 yield more consistent interpretation compared to low-performing models. Finally, AIOps models that are constructed with the Sliding Window or Full History approaches have the most consistent interpretation with the trends presented in the entire datasets. Our study provides valuable guidelines for practitioners to derive consistent AIOps model interpretation.

show abstract

“…We use a linear Support Vector Machine (SVM) with bagof-words features based on n-grams as a baseline for VulDeePecker (see Appendix C for details). To see what VulDeePecker has learned we follow the work of Warnecke et al [129] and use the Layerwise Relevance Propagation (LRP) method [12] to explain the predictions and assign each token a relevance score that indicates its importance for the classification. Results.…”

Section: Vulnerability Discoverymentioning

confidence: 99%

Dos and Don'ts of Machine Learning in Computer Security

Arp¹,

Quiring²,

Pendlebury³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

With the growing processing power of computing systems and the increasing availability of massive datasets, machine learning algorithms have led to major breakthroughs in many different areas. This development has influenced computer security, spawning a series of work on learning-based security systems, such as for malware detection, vulnerability discovery, and binary code analysis. Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance and render learning-based systems potentially unsuitable for security tasks and practical deployment.In this paper, we look at this problem with critical eyes. First, we identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We conduct a longitudinal study of 30 papers from top-tier security conferences within the past 10 years, confirming that these pitfalls are widespread in the current security literature. In an empirical analysis, we further demonstrate how individual pitfalls can lead to unrealistic performance and interpretations, obstructing the understanding of the security problem at hand. As a remedy, we derive a list of actionable recommendations to support researchers and our community in avoiding pitfalls, promoting a sound design, development, evaluation, and deployment of learning-based systems for computer security.

show abstract

Evaluating Explanation Methods for Deep Learning in Security

Cited by 78 publications

References 23 publications

Deep security analysis of program code

Deep security analysis of program code

Towards a Consistent Interpretation of AIOps Models

Dos and Don'ts of Machine Learning in Computer Security

Contact Info

Product

Resources

About