Safe-visor architecture for sandboxing (AI-based) unverified controllers in stochastic cyber–physical systems

Zhong, Bingzhuo; Lavaei, Abolfazl; Cao, Hongpeng; Caccamo, Marco

doi:10.1016/j.nahs.2021.101110

Cited by 7 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cheng et al [3] use action projection and train a second model on the previous interventions to reduce the need for future interventions. Zhong et al [15] derive a safe-visor that rejects infeasible actions proposed by the agent and replaces it with a safe action.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Generate All Feasible Actions

Theile,

Bernardini,

Trumpp

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Modern cyber-physical systems are becoming increasingly complex to model, thus motivating data-driven techniques such as reinforcement learning (RL) to find appropriate control agents. However, most systems are subject to hard constraints such as safety or operational bounds. Typically, to learn to satisfy these constraints, the agent must violate them systematically, which is computationally prohibitive in most systems. Recent efforts aim to utilize feasibility models that assess whether a proposed action is feasible to avoid applying the agent's infeasible action proposals to the system. However, these efforts focus on guaranteeing constraint satisfaction rather than the agent's learning efficiency. To improve the learning process, we introduce action mapping, a novel approach that divides the learning process into two steps: first learn feasibility and subsequently, the objective by mapping actions into the sets of feasible actions. This paper focuses on the feasibility part by learning to generate all feasible actions through self-supervised querying of the feasibility model. We train the agent by formulating the problem as a distribution matching problem and deriving gradient estimators for different divergences. Through an illustrative example, a robotic path planning scenario, and a robotic grasping simulation, we demonstrate the agent's proficiency in generating actions across disconnected feasible action sets. By addressing the feasibility step, this paper makes it possible to focus future work on the objective part of action mapping, paving the way for an RL framework that is both safe and efficient.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Using the actions a i as support of the KDE in (12), the densities qθ,σ (a * j ) and qθ,σ ′ (a * j ) are computed. Then the feasibility model g is evaluated on all samples a * j and the estimate of p(a * j ) is computed using (5) and importance sampling in (15). Finally, the gradient of θ can be computed according to (14).…”

Section: Training Processmentioning

confidence: 99%

Learning to Generate All Feasible Actions

Theile,

Bernardini,

Trumpp

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…Explainable AI (XAI) focuses upon improving the transparency of AI decision-making processes, to provide clarity and justification to actions such as those that result in undesirable behaviour [4,5]. Publications in AI Safety include pragmatic approaches for harm avoidance and self-supervisory wrapper systems [6,7] as well as social approaches including exploration of legal regulation [8]. Recent work in Impact Minimisation (IM) seeks to generalise and penalise against any impactful behaviours that are not explicitly aligned with the agent's primary objective [9,10].…”

Section: Introductionmentioning

confidence: 99%

AI apology: interactive multi-objective reinforcement learning for human-aligned AI

Harland

Dazeley

Nakisa

et al. 2023

Neural Comput & Applic

View full text Add to dashboard Cite

For an Artificially Intelligent (AI) system to maintain alignment between human desires and its behaviour, it is important that the AI account for human preferences. This paper proposes and empirically evaluates the first approach to aligning agent behaviour to human preference via an apologetic framework. In practice, an apology may consist of an acknowledgement, an explanation and an intention for the improvement of future behaviour. We propose that such an apology, provided in response to recognition of undesirable behaviour, is one way in which an AI agent may both be transparent and trustworthy to a human user. Furthermore, that behavioural adaptation as part of apology is a viable approach to correct against undesirable behaviours. The Act-Assess-Apologise framework potentially could address both the practical and social needs of a human user, to recognise and make reparations against prior undesirable behaviour and adjust for the future. Applied to a dual-auxiliary impact minimisation problem, the apologetic agent had a near perfect determination and apology provision accuracy in several non-trivial configurations. The agent subsequently demonstrated behaviour alignment with success that included up to complete avoidance of the impacts described by these objectives in some scenarios.

show abstract

“…Various formal verification and synthesis techniques have been investigated to ensure safety in CPS [4][5][6][7]. Abstraction-based methods have gained significant popularity in the last two decades for safety analysis of CPS [6][7][8][9][10]. These methods approximate original systems with continuous state and input sets by their finite abstractions, constructed by discretizing the original sets.…”

Section: Introductionmentioning

confidence: 99%

Synthesizing Safety Controllers for Uncertain Linear Systems: A Direct Data-driven Approach

Zhong

Caccamo

2022

2022 IEEE Conference on Control Technology and Applications (CCTA)

View full text Add to dashboard Cite

In this paper, we present the synthesis of secure-by-construction controllers that address safety and security properties simultaneously in cyber-physical systems. Our focus is on studying a specific security property called opacity, which characterizes the system's ability to maintain plausible deniability of its secret behavior in the presence of an intruder. These controllers are synthesized based on a concept of so-called (augmented) control barrier functions, which we introduce and discuss in detail. We propose conditions that facilitate the construction of the desired (augmented) control barrier functions and their corresponding secure-by-construction controllers. To compute these functions, we propose an iterative scheme that leverages iterative sum-of-square programming techniques. This approach enables efficient computation of these functions, particularly for polynomial systems. Moreover, we demonstrate the flexibility of our approach by incorporating user-defined cost functions into the construction of secure-by-construction controllers. Finally, we validate the effectiveness of our results through two case studies, illustrating the practical applicability and benefits of our proposed approach.

show abstract

Safe-visor architecture for sandboxing (AI-based) unverified controllers in stochastic cyber–physical systems

Cited by 7 publications

References 15 publications

Learning to Generate All Feasible Actions

Learning to Generate All Feasible Actions

AI apology: interactive multi-objective reinforcement learning for human-aligned AI

Synthesizing Safety Controllers for Uncertain Linear Systems: A Direct Data-driven Approach

Contact Info

Product

Resources

About