Robustness to Adversarial Attacks in Learning-Enabled Controllers

Xiong, Zikang; Eappen, Joe; Zhu, He; Jagannathan, Suresh

doi:10.48550/arxiv.2006.06861

Cited by 4 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The second category of explainable transition and reward models leverages understandable models of the task or environment, e.g., a transition model (Martínez et al, 2016;Zhu et al, 2020) or a preference model (Icarte et al, 2018;Icarte et al, 2019). Such models help explain both the RL agent's reasoning about its decision-making and humans' understanding of the decision-making process.…”

Section: Interpretable/explainable Decision-making Processes Of Rlmentioning

confidence: 99%

“…However, while testing the model is an essential part of the training loop, it should not be the only component to ensure safe operation. An example of how safety can be ensured is presented by Xiong et al (2020), who propose using shield-based defenses, where agents learn to stay within predefined, safe boundaries during training and application, and with that increase robustness.…”

Section: Focus: Safety Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Retzlaff,

Das,

Wayllace

et al. 2024

jair

View full text Add to dashboard Cite

Artificial intelligence (AI) and especially reinforcement learning (RL) have the potential to enable agents to learn and perform tasks autonomously with superhuman performance. However, we consider RL as fundamentally a Human-in-the-Loop (HITL) paradigm, even when an agent eventually performs its task autonomously. In cases where the reward function is challenging or impossible to define, HITL approaches are considered particularly advantageous. The application of Reinforcement Learning from Human Feedback (RLHF) in systems such as ChatGPT demonstrates the effectiveness of optimizing for user experience and integrating their feedback into the training loop. In HITL RL, human input is integrated during the agent’s learning process, allowing iterative updates and fine-tuning based on human feedback, thus enhancing the agent’s performance. Since the human is an essential part of this process, we argue that human-centric approaches are the key to successful RL, a fact that has not been adequately considered in the existing literature. This paper aims to inform readers about current explainability methods in HITL RL. It also shows how the application of explainable AI (xAI) and specific improvements to existing explainability approaches can enable a better human-agent interaction in HITL RL for all types of users, whether for lay people, domain experts, or machine learning specialists. Accounting for the workflow in HITL RL and based on software and machine learning methodologies, this article identifies four phases for human involvement for creating HITL RL systems: (1) Agent Development, (2) Agent Learning, (3) Agent Evaluation, and (4) Agent Deployment. We highlight human involvement, explanation requirements, new challenges, and goals for each phase. We furthermore identify low-risk, high-return opportunities for explainability research in HITL RL and present long-term research goals to advance the field. Finally, we propose a vision of human-robot collaboration that allows both parties to reach their full potential and cooperate effectively.

show abstract

Section: Interpretable/explainable Decision-making Processes Of Rlmentioning

confidence: 99%

Section: Focus: Safety Evaluationmentioning

confidence: 99%

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Retzlaff,

Das,

Wayllace

et al. 2024

jair

View full text Add to dashboard Cite

show abstract

“…Clark et al [113] also investigated the impact of adversarial attacks on the ML policies for controlling a robotic system. Finally, Xiong et al in [114] proposed attacks and defenses against ML algorithms used in learning-enabled controllers.…”

Section: Adversarial Machine Learning (Aml) and Cpsmentioning

confidence: 99%

Resilient Machine Learning for Networked Cyber Physical Systems: A Survey for Machine Learning Security to Securing Machine Learning for CPS

Olowononi

Rawat

Liu

2021

IEEE Commun. Surv. Tutorials

146

View full text Add to dashboard Cite

Cyber Physical Systems (CPS) are characterized by their ability to integrate the physical and information or cyber worlds. Their deployment in critical infrastructure have demonstrated a potential to transform the world. However, harnessing this potential is limited by their critical nature and the far reaching effects of cyber attacks on human, infrastructure and the environment. An attraction for cyber concerns in CPS rises from the process of sending information from sensors to actuators over the wireless communication medium, thereby widening the attack surface. Traditionally, CPS security has been investigated from the perspective of preventing intruders from gaining access to the system using cryptography and other access control techniques. Most research work have therefore focused on the detection of attacks in CPS. However, in a world of increasing adversaries, it is becoming more difficult to totally prevent CPS from adversarial attacks, hence the need to focus on making CPS resilient. Resilient CPS are designed to withstand disruptions and remain functional despite the operation of adversaries. One of the dominant methodologies explored for building resilient CPS is dependent on machine learning (ML) algorithms. However, rising from recent research in adversarial ML, we posit that ML algorithms for securing CPS must themselves be resilient. This paper is therefore aimed at comprehensively surveying the interactions between resilient CPS using ML and resilient ML when applied in CPS. The paper concludes with a number of research trends and promising future research directions. Furthermore, with this paper, readers can have a thorough understanding of recent advances on ML-based security and securing ML for CPS and countermeasures, as well as research trends in this active research area.

show abstract

“…We highlight that our method can handle any other open-loop or feedback controllers that are not necessarily modeled as NNs. In this paper, we consider NN controllers due to their brittleness to small input perturbations [40], [41] and their lack of safe generalization to unseen tasks.…”

Section: Introductionmentioning

confidence: 99%

Verified Compositions of Neural Network Controllers for Temporal Logic Control Objectives

Wang

Kalluraya

Kantaros

2022

2022 IEEE 61st Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

Several methods have been proposed recently to learn neural network (NN) controllers for autonomous agents, with unknown and stochastic dynamics, tasked with complex missions captured by Linear Temporal Logic (LTL). Due to the sample-inefficiency of the majority of these works, compositional learning methods have been proposed decomposing the LTL specification into smaller sub-tasks. Then, separate controllers are learned and composed to satisfy the original task. A key challenge within these approaches is that they often lack safety guarantees or the provided guarantees are impractical. This paper aims to address this challenge. Particularly, we consider autonomous systems with unknown and stochastic dynamics and LTL-encoded tasks. We assume that the system is equipped with a finite set of base skills modeled by trained NN feedback controllers. Our goal is to check if there exists a temporal composition of the trained NN controllers -and if so, to compute it -that will yield a composite system behavior that satisfies the assigned LTL task with probability one. We propose a new approach that relies on a novel integration of automata theory and data-driven reachability analysis tools for NN-controlled stochastic systems. The resulting neuro-symbolic controller allows the agent to generate safe behaviors for unseen complex temporal logic tasks in a zero-shot fashion by leveraging its base skills. We show correctness of the proposed method and we provide conditions under which it is complete. To the best of our knowledge, this is the first work that designs verified temporal compositions of NN controllers for unknown and stochastic systems. Finally, we provide extensive numerical simulations and hardware experiments on robot navigation tasks to demonstrate the proposed method.

show abstract

Robustness to Adversarial Attacks in Learning-Enabled Controllers

Cited by 4 publications

References 19 publications

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Resilient Machine Learning for Networked Cyber Physical Systems: A Survey for Machine Learning Security to Securing Machine Learning for CPS

Verified Compositions of Neural Network Controllers for Temporal Logic Control Objectives

Contact Info

Product

Resources

About