Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning

Lavaei, Abolfazl; Somenzi, Fabio; Soudjani, Sadegh; Trivedi, Ashutosh

doi:10.1109/iccps48487.2020.00017

Cited by 38 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement learning [186] (RL) is a samplingbased optimization algorithm that computes optimal policies driven by scalar reward signals. Recently, RL has been extended to work with formal logic [32,33,138,68,98], and automatic structures (ω-automata [64,65] and reward machines [77]) instead of scalar reward signals. A promising future direction is to extend RL-based synthesis to reason with security properties of the system.…”

Section: Future Directionsmentioning

confidence: 99%

“…In such systems, security verification need to reason with neural networks along with the system dynamics. There is a large body of work [74,1,64,152,115,209,98] in verifying control systems with neural networks using SMT solvers, and will provide a promising avenue of research in developing security verification and synthesis approaches for CPS with neural networks based controllers.…”

Section: Future Directionsmentioning

confidence: 99%

See 1 more Smart Citation

Secure-by-Construction Synthesis of Cyber-Physical Systems

Liu¹,

Trivedi²,

Yin³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Correct-by-construction synthesis is a cornerstone of the confluence of formal methods and control theory towards designing safety-critical systems. Instead of following the time-tested, albeit laborious (re)design-verify-validate loop, correct-by-construction methodology advocates the use of continual refinements of formal requirements-connected by chains of formal proofs-to build a system that assures the correctness by design. A remarkable progress has been made in scaling the scope of applicability of correct-byconstruction synthesis-with a focus on cyber-physical systems that tie discrete-event control with continuous environment-to enlarge control systems by combining symbolic approaches with principled state-space reduction techniques. Unfortunately, in the security-critical control systems, the security properties are verified ex post facto the design process in a way that undermines the correct-by-construction paradigm. We posit that, to truly realize the dream of correct-by-construction synthesis for security-critical systems, security considerations must take center-stage with the safety considerations. Moreover, catalyzed by the recent progress on the opacity sub-classes of security properties and the notion of hyperproperties capable of combining security with safety properties, we believe that the time is ripe for the research community to holistically target the challenge of secure-by-construction synthesis. This paper details our vision by highlighting the recent progress and open challenges that may serve as bricks for providing a solid foundation for secure-by-construction synthesis of cyber-physical systems.The revolution in miniaturized communication devices in the beginning of this millennium contributed towards a revolution in the internet-of-things (IoT) and the networked systems woven around them: the cyber-physical systems (CPS). CPS are marked by a close-knit interaction of discrete computation and continuous control over a network and are playing critical roles in virtually every aspect of our modern experience ranging from consumer electronics to implantable medical devices, from smart cars to smart hospitals, and from controlling our power systems to safeguarding our nuclear rectors. These systems are clearly safety-critical as a bug in their design could be life threatening, but given their societal implications, they are also security-critical where a bug in their design may have the potential to jeopardize the privacy, trust, and economic interests of society built around them." We believe that the security considerations should be elevated as primary design drivers along with safety ones to tackle the design challenge of modern CPS and call for a need to expand the correct-byconstruction paradigm of designing safety-critical systems to encompass security: we call this paradigm secure-by-construction. This paper synthesizes ideas from three research communities: discrete event systems (DES), control systems (CS), and formal methods (FM) to pose and study central problems supporting...

show abstract

Section: Future Directionsmentioning

confidence: 99%

Section: Future Directionsmentioning

confidence: 99%

Secure-by-Construction Synthesis of Cyber-Physical Systems

Liu¹,

Trivedi²,

Yin³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Techniques from formal methods can be effectively used to address the problem of principled history-aware reward engineering. For instance, automata-based approaches [31,19,17,25] have been used to solve the issue of non-Markovian objectives for omega-regular objectives for infinite behavior. However, these methods are susceptible to the definition of sparse reward functions.…”

Section: Introductionmentioning

confidence: 99%

Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives

Balakrishnan¹,

Jakšić²,

Aguilar³

et al. 2022

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) is a popular approach for robotic path planning in uncertain environments. However, the control policies trained for an RL agent crucially depend on user-defined, state-based reward functions. Poorly designed rewards can lead to policies that do get maximal rewards but fail to satisfy desired task objectives or are unsafe. There are several examples of the use of formal language such as temporal logics and automata to specify high-level task specifications for robots (in lieu of Markovian rewards). Recent efforts have focused on inferring state-based rewards from formal specifications; here, the goal is to provide (probabilistic) guarantees that the policy learned using RL (with the inferred rewards) satisfies the highlevel formal specification. A key drawback of several of these techniques is that the rewards that they infer are sparse: the agent receives positive rewards only upon completion of the task and no rewards otherwise. This naturally leads to poor convergence properties and high variance during RL.In this work we propose using formal specifications in the form of symbolic automata: these serve as a generalization of both bounded-time temporal logic based specifications as well as automata. Furthermore our use of symbolic automata allows us to define non-sparse potential-based rewards which empirically shape the reward surface, leading to better convergence during RL. We also show that our potential-based rewarding strategy still allows us to obtain the policy that maximizes the satisfaction of the given specification.

show abstract

“…We, however, focus on (infinite-state) stochastic hybrid systems in continuous time. In the discrete-time setting, Lavaei et al [31] recently proposed a way to learn on an abstraction of the continuous state space that provides guarantees on the error of the final results. Other than that, learning for these systems-in particular in continuous time-has not been well-explored from a formal perspective w.r.t.…”

Section: Introductionmentioning

confidence: 99%

Learning optimal decisions for stochastic hybrid systems

Niehage

Hartmanns

Remke

2021

Proceedings of the 19th ACM-IEEE International Conference on Formal Methods and Models for System Design

View full text Add to dashboard Cite

We apply reinforcement learning to approximate the optimal probability that a stochastic hybrid system satisfies a temporal logic formula. We consider systems with (non)linear continuous dynamics, random events following general continuous probability distributions, and discrete nondeterministic choices. We present a discretized view of states to the learner, but simulate the continuous system. Once we have learned a near-optimal scheduler resolving the choices, we use statistical model checking to estimate its probability of satisfying the formula. We implemented the approach using Q-learning in the tools HYPEG and modes, which support Petri net-and hybrid automata-based models, respectively. Via two case studies, we show the feasibility of the approach, and compare its performance and effectiveness to existing analytical techniques for a linear model. We find that our new approach quickly finds near-optimal prophetic as well as non-prophetic schedulers, which maximize or minimize the probability that a specific signal temporal logic property is satisfied.

show abstract

Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning

Cited by 38 publications

References 27 publications

Secure-by-Construction Synthesis of Cyber-Physical Systems

Secure-by-Construction Synthesis of Cyber-Physical Systems

Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives

Learning optimal decisions for stochastic hybrid systems

Contact Info

Product

Resources

About