Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Iyer, Rashi; Li, Yuezhang; Li, Huao; Lewis, Michael; Sundar, R.; Sycara, Katia

doi:10.48550/arxiv.1809.06061

Cited by 3 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, model-agnostic explanation methods usually work by analyzing feature inputs and outputs and do not have access to the models' internal information, such as weights or structural information by definition. Shapley Additive Explanations (SHAP) tools [59], Saliency Map [60], and Gradient-weighted Class Activation Mapping (Grad-CAM) [61] are widely used model-agnostic explanation tools.…”

Section: ) Model-specific or Model-agnosticmentioning

confidence: 99%

Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research

et al. 2022

View full text Add to dashboard Cite

This survey presents a comprehensive review of current literature on Explainable Artificial Intelligence (XAI) methods for cyber security applications. Due to the rapid development of Internetconnected systems and Artificial Intelligence in recent years, Artificial Intelligence including Machine Learning (ML) and Deep Learning (DL) has been widely utilized in the fields of cyber security including intrusion detection, malware detection, and spam filtering. However, although Artificial Intelligence-based approaches for the detection and defense of cyber attacks and threats are more advanced and efficient compared to the conventional signature-based and rule-based cyber security strategies, most ML-based techniques and DL-based techniques are deployed in the ''black-box'' manner, meaning that security experts and customers are unable to explain how such procedures reach particular conclusions. The deficiencies of transparencies and interpretability of existing Artificial Intelligence techniques would decrease human users' confidence in the models utilized for the defense against cyber attacks, especially in current situations where cyber attacks become increasingly diverse and complicated. Therefore, it is essential to apply XAI in the establishment of cyber security models to create more explainable models while maintaining high accuracy and allowing human users to comprehend, trust, and manage the next generation of cyber defense mechanisms. Although there are papers reviewing Artificial Intelligence applications in cyber security areas and the vast literature on applying XAI in many fields including healthcare, financial services, and criminal justice, the surprising fact is that there are currently no survey research articles that concentrate on XAI applications in cyber security. Therefore, the motivation behind the survey is to bridge the research gap by presenting a detailed and up-to-date survey of XAI approaches applicable to issues in the cyber security field. Our work is the first to propose a clear roadmap for navigating the XAI literature in the context of applications in cyber security.

show abstract

Section: ) Model-specific or Model-agnosticmentioning

confidence: 99%

Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research

et al. 2022

View full text Add to dashboard Cite

show abstract

“…LRP has been used to show what information is important to the agent towards a specific decision: The approach presented in [64] addresses the policy and the responses explanation problems by combining methods, contributing to our understanding of the role of saliency maps in the context of explaining agents' behaviour: While saliency maps have been shown to improve classification decisions in images (albeit with 60% of correctness in [68]), their role in RL is not that clear (e.g. as shown in [3]). However, they may provide significant positive effects when combined with other methods (as in [69]).…”

Section: Lime and Shapley Sampling Values Show Improvement In Terms O...mentioning

confidence: 99%

“…4.4.5 Understanding agent actions using specific and relevant feature attribution. Recognizing that perturbation-based approaches for RL, as that proposed by S. Greydanus et al in [66] and the O-DRL approach [3] in 4.4.4, tend to produce saliency maps that are not specific to the action of interest, N. Puri et al in [63] propose SARFA to generate saliency maps that focus on explaining the specific action decided, balancing between specificity and relevance.…”

Section: Transparencymentioning

confidence: 99%

Section: Transparencymentioning

confidence: 99%

“…SAFRA in [63] has been evaluated qualitatively in games (chess, Attari games and Go), showing that in comparison to O-DRL [3] and the approach in [66] produces more focused saliency maps. In addition, human studies on problemsolving chess puzzles show that saliency maps produced by SAFRA are less confusing than those produced by [3] and [66]. SAFRA is compared to the other approaches using a chess saliency dataset produced by human experts, showing superiority in identifying salient features and robustness in pertrubations.…”

Section: Transparencymentioning

confidence: 99%

See 2 more Smart Citations

Explainable Deep Reinforcement Learning: State of the Art and Challenges

Vouros

2022

ACM Comput. Surv.

View full text Add to dashboard Cite

Interpretability, explainability and transparency are key issues to introducing Artificial Intelligence methods in many critical domains: This is important due to ethical concerns and trust issues strongly connected to reliability, robustness, auditability and fairness, and has important consequences towards keeping the human in the loop in high levels of automation, especially in critical cases for decision making, where both (human and the machine) play important roles. While the research community has given much attention to explainability of closed (or black) prediction boxes, there are tremendous needs for explainability of closed-box methods that support agents to act autonomously in the real world. Reinforcement learning methods, and especially their deep versions, are such closed-box methods. In this article we aim to provide a review of state of the art methods for explainable deep reinforcement learning methods, taking also into account the needs of human operators - i.e., of those that take the actual and critical decisions in solving real-world problems. We provide a formal specification of the deep reinforcement learning explainability problems, and we identify the necessary components of a general explainable reinforcement learning framework. Based on these, we provide a comprehensive review of state of the art methods, categorizing them in classes according to the paradigm they follow, the interpretable models they use, and the surface representation of explanations provided. The article concludes identifying open questions and important challenges.

show abstract

Bridging the Gap: Advancing the Transparency and Trustworthiness of Network Intrusion Detection with Explainable AI

Islam,

Syfullah,

Rashed

et al. 2023

Preprint

View full text Add to dashboard Cite

With the explosive rise of internet usage and the development of web applications across various platforms, ensuring network and system security has become a critical concern. Networks and web services are particularly susceptible to targeted attacks, as hackers and intruders persistently attempt to gain unauthorized access. The integration of artificial intelligence (AI) has emerged as a crucial tool for detecting intrusions and constructing effective Intrusion Detection Systems (IDSs) to counter cyber-attacks and malicious activities. IDSs developed using machine learning (ML) and deep learning (DL) techniques have proven to be highly effective in detecting network attacks, offering machine-centric solutions. Nevertheless, mainstream adoption, confidence and trust in these systems have been greatly impeded by the fact that ML/DL implementations tend to be “black boxes,” and thus lacking human interpretability, transparency, explainability, and logical reasoning in their prediction outputs. This limitation has prompted questions about the responsibility and comprehension of AI-driven intrusion detection systems. In this study, we propose four novel architectures that incorporate Explainable Artificial Intelligence (XAI) techniques to overcome the challenges of limited interpretability in ML/DL based IDSs. We focus on the development of ExplainDTC, SecureForest-RFE, RationaleNet, and CNNShield architectures in network security solutions, and inquiry into their potential to convert the untrustworthy architectures into trustworthy. The models are applied to scan network traffic and identify, and report intrusions based on the traits extracted from the UNSW-NB15 dataset. To explain how a decision is made by the models and to add expansibility at every stage of machine learning pipeline, we integrate multiple XAI methods such as LIME, SHAP, ElI5, and ProtoDash on top of our architectures. The generated explanations provide quantifiable insights into the influential factors and their respective impact on network intrusion predictions.

show abstract

Transparency and Explanation in Deep Reinforcement Learning Neural Networks

Cited by 3 publications

References 0 publications

Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research

Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research

Explainable Deep Reinforcement Learning: State of the Art and Challenges

Bridging the Gap: Advancing the Transparency and Trustworthiness of Network Intrusion Detection with Explainable AI

Contact Info

Product

Resources

About