A Safety Aware Model-Based Reinforcement Learning Framework for Systems with Uncertainties

Mahmud, Shakil; Hareland, Katrine; Nivison, Scott; Bell, Zachary I.; Kamalapurkar, Rushikesh

doi:10.23919/acc50511.2021.9482976

Cited by 11 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, ∇B f ∈ L ∞ . Since all constituents of the control law in (15) are bounded, we can conclude that û(t) ∈ L ∞ . Remark 3: Theorem 1 proves that the control policy in (15) guarantees safety for all time.…”

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 95%

“…Since all constituents of the control law in (15) are bounded, we can conclude that û(t) ∈ L ∞ . Remark 3: Theorem 1 proves that the control policy in (15) guarantees safety for all time. Further, the control policy doesn't switch between a stabilizing backup policy and the RL policy, which is a distinct advantage over approaches that rely on an elusive backup policy.…”

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 95%

“…Theorem 1: Under the control law in (15) and provided Assumptions 1-4 hold, the set C is forward invariant for the system in (1) if x(0) ∈ C.…”

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 99%

“…where ϕ 1 was defined in (23). Theorem 3: Provided Assumptions 1-5 hold, the regressor matrix ψ(t) is PE, and the following gain conditions c 3 < η a1 k a k s and 4η a2 > k s are satisfied, then the control action in (15), the actor and critic update laws from ( 21) and ( 24), and the identifier (26), ( 27) guarantee that the state x(t), actor weight estimation error Wa (t) and the critic weight estimation error Wc (t) are UUB.…”

Section: Identifier Designmentioning

confidence: 99%

“…The safety problem of continuous-time RL is primarily addressed by considering the continuous-time counterpart of CBFs, namely barrier Lyapunov functions (BLF) [12]. One research direction is to transform the constrained state dynamics into dynamics of an unconstrained state [13]- [15] and subsequently use ADP algorithms to solve the unconstrained problem. However, this approach typically handles rectangular state constraints (box constraints on individual components of states) and cannot be trivially extended to general convex state constraints.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

Bandyopadhyay¹,

Bhasin²

2022

Preprint

View full text Add to dashboard Cite

This paper proposes an on-policy reinforcement learning (RL) control algorithm that solves the optimal regulation problem for a class of uncertain continuous-time nonlinear systems under user-defined state constraints. We formulate the safe RL problem as the minimization of the Hamiltonian subject to a constraint on the time-derivative of a barrier Lyapunov function (BLF). We subsequently use the analytical solution of the optimization problem to modify the Actor-Critic-Identifier architecture to learn the optimal control policy safely. The proposed method does not require the presence of external backup controllers, and the RL policy ensures safety for the entire duration. The efficacy of the proposed controller is demonstrated on a class of Euler-Lagrange systems.

show abstract

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 95%

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 95%

“…Theorem 1: Under the control law in (15) and provided Assumptions 1-4 hold, the set C is forward invariant for the system in (1) if x(0) ∈ C.…”

Section: B Blf-based Constrained Optimal Control Problemmentioning

confidence: 99%

Section: Identifier Designmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

Bandyopadhyay¹,

Bhasin²

2022

Preprint

View full text Add to dashboard Cite

show abstract

Safe adaptive output‐feedback optimal control of a class of linear systems

Mahmud,

Abudia,

Nivison

et al. 2024

Intl J Robust & Nonlinear

View full text Add to dashboard Cite

The objective of this research is to enable safety‐critical systems to simultaneously learn and execute optimal control policies in a safe manner to achieve complex autonomy. Learning optimal policies via trial and error, that is, traditional reinforcement learning, is difficult to implement in safety‐critical systems, particularly when task restarts are unavailable. Safe model‐based reinforcement learning techniques based on a barrier transformation have recently been developed to address this problem. However, these methods rely on full‐state feedback, limiting their usability in a real‐world environment. In this work, an output‐feedback safe model‐based reinforcement learning technique based on a novel barrier‐aware dynamic state estimator has been designed to address this issue. The developed approach facilitates simultaneous learning and execution of safe control policies for safety‐critical linear systems. Simulation results indicate that barrier transformation is an effective approach to achieve online reinforcement learning in safety‐critical systems using output feedback.

show abstract

Machine learning model‐based optimal tracking control of nonlinear affine systems with safety constraints

Wang,

2024

Intl J Robust & Nonlinear

View full text Add to dashboard Cite

This work focuses on the development of a machine learning (ML) model‐based framework for safe optimal tracking control of a class of nonlinear control‐affine systems to ensure simultaneous closed‐loop stability and safety. Specifically, a novel multilayer feedforward neural network (FNN) with a control‐affine architecture is designed to model nonlinear dynamic systems. Subsequently, a model‐based reinforcement learning (RL) framework is presented, utilizing a novel cost function with Control Lyapunov‐Barrier Function (CLBF) properties, to learn both the control policy and the optimal value function for an infinite‐horizon optimal tracking control problem for nonlinear systems with safety constraints. The efficacy of the proposed methodology is demonstrated through simulations of a one‐link robot manipulator and a chemical process example.

show abstract

A Safety Aware Model-Based Reinforcement Learning Framework for Systems with Uncertainties

Cited by 11 publications

References 22 publications

Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

Safe reinforcement learning control for continuous-time nonlinear systems without a backup controller

Safe adaptive output‐feedback optimal control of a class of linear systems

Machine learning model‐based optimal tracking control of nonlinear affine systems with safety constraints

Contact Info

Product

Resources

About