Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions

Choi, Tsan‐Ming; Castañeda, Fernando; Tomlin, Claire J.; Sreenath, Koushil

doi:10.48550/arxiv.2004.07584

Cited by 30 publications

(43 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Stability Guarantees For Neural Network Controlled Systems. As neural networks become popular in control tasks, safety and robustness of neural networks and neural network controlled systems has been actively discussed (Morimoto and Doya 2005;Luo, Wu, and Huang 2014;Friedrich and Buss 2017;Berkenkamp et al 2017;Chow et al 2018;Matni et al 2019;Han et al 2019;Recht 2019;Choi et al 2020;Zhang, Hu, and Basar 2020;Fazlyab, Morari, and Pappas 2020). Closely related to this work are recent papers on robustness analysis of memory-less neural networks controlled systems based on robust control ideas.…”

Section: Related Workmentioning

confidence: 90%

Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems

Gu¹,

Yin²,

Ghaoui³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural network controllers have become popular in control tasks thanks to their flexibility and expressivity. Stability is a crucial property for safety-critical dynamical systems, while stabilization of partially observed systems, in many cases, requires controllers to retain and process long-term memories of the past. We consider the important class of recurrent neural networks (RNN) as dynamic controllers for nonlinear uncertain partially-observed systems, and derive convex stability conditions based on integral quadratic constraints, S-lemma and sequential convexification. To ensure stability during the learning and control process, we propose a projected policy gradient method that iteratively enforces the stability conditions in the reparametrized space taking advantage of mild additional information on system dynamics. Numerical experiments show that our method learns stabilizing controllers while using fewer samples and achieving higher final performance compared with policy gradient.

show abstract

Section: Related Workmentioning

confidence: 90%

Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems

Gu¹,

Yin²,

Ghaoui³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…CBF can be constructed empirically [35] or learned from data [36], [37]. Along with optimization-based control actions or reinforcement learning (RL) techniques, the CBFbased approach to ensure safety has achieved satisfying results in various application scenarios, including bipedal locomotion under model uncertainty [38], autonomous vehicles [39], and UAVs [40]. Recently, CBF techniques were generalized to stochastic systems with high-probability guarantees, in cases of both complete and incomplete information in [41] and [42].…”

Section: Introductionmentioning

confidence: 99%

Generalization of Safe Optimal Control Actions on Networked Multi-Agent Systems

Song¹,

Wan²,

Gahlawat³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a unified framework to fast generate a safe optimal control action for a new task from existing controllers on Multi-Agent Systems (MASs). The control action composition is achieved by taking a weighted mixture of the existing controllers according to the contribution of each component task. Instead of sophisticatedly tuning the cost parameters and other hyper-parameters for safe and reliable behavior in the optimal control framework, the safety of each single task solution is guaranteed using the control barrier functions (CBFs) for high-degree stochastic systems, which constrains the system state within a known safe operation region where it originates from. Linearity of CBF constraints in control enables the control action composition. The discussed framework can immediately provide reliable solutions to new tasks by taking a weighted mixture of solved component-task actions and filtering on some CBF constraints, instead of performing an extensive sampling to achieve a new controller. Our results are verified and demonstrated on both a single UAV and two cooperative UAV teams in an environment with obstacles.

show abstract

“…As such, the proposed method is applicable to large scale systems. Another similar work was presented in [Choi et al, 2020] where a unified RL-based framework was used to learn the dynamic uncertainty in the control Lyapunov function, control Barrier function, and other dynamic control affine constraints altogether in a single learning process. Again, the distinction of our work is that the Barrier function was included in the reward function to guide the control learning process, without any model knowledge.…”

Section: Introductionmentioning

confidence: 99%

Safe Reinforcement Learning for Grid Voltage Control

Mukherjee

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

Under voltage load shedding has been considered as a standard approach to recover the voltage stability of the electric power grid under emergency conditions; yet this scheme usually trips a massive amount of load inefficiently. Reinforcement learning (RL) has been adopted as a promising approach to circumvent the issues; however, RL approach usually cannot guarantee the safety of the systems under control. In this paper, we discuss a couple of novel safe RL approaches, namely constrained optimization approach and Barrier function-based approach, that can safely recover voltage under emergency events. This method is general and can be applied to other safety-critical control problems. Numerical simulations on the 39-bus IEEE benchmark are performed to demonstrate the effectiveness of the proposed safe RL emergency control.

show abstract

Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions

Cited by 30 publications

References 0 publications

Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems

Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems

Generalization of Safe Optimal Control Actions on Networked Multi-Agent Systems

Safe Reinforcement Learning for Grid Voltage Control

Contact Info

Product

Resources

About