Relaxed Controls and the Convergence of Optimal Control Algorithms

Williamson, Lorna J.; Polak, E.

doi:10.1137/0314047

Cited by 38 publications

(3 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To consider a maximum entropy variant of the optimal control problem, we now generalize the notion of controls by taking the relaxed control approach. This approach was first introduced by Young [59,60], and then widely applied to calculus of variations [36,53], deterministic optimal control [1,54,55] and stochastic optimal control [6,20,25]. Consider a function µ : [0, T ] → P(U ).…”

Section: Problem Setupmentioning

confidence: 99%

Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

Kim,

Yang

2020

Preprint

View full text Add to dashboard Cite

Maximum entropy reinforcement learning (RL) methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of existing techniques are designed for discrete-time systems. As a first step toward their extension to continuous-time systems, this paper considers continuous-time deterministic optimal control problems with entropy regularization. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. Our maximum entropy formulation is shown to enhance the regularity of the viscosity solution and to be asymptotically consistent as the effect of entropy regularization diminishes. A salient feature of the HJB equations is computational tractability. Generalized Hopf-Lax formulas can be used to solve the HJB equations in a tractable grid-free manner without the need for numerically optimizing the Hamiltonian. We further show that the optimal control is uniquely characterized as Gaussian in the case of control affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. Lastly, we discuss how to extend our results to continuous-time model-free RL by taking an adaptive dynamic programming approach. To our knowledge, the resulting algorithms are the first data-driven control methods that use an information theoretic exploration mechanism in continuous time.

show abstract

Section: Problem Setupmentioning

confidence: 99%

Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

Kim,

Yang

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the existence of Lagrange Multipliers is also important in order to prove the convergence of computational procedures (cf. Wierzbicki/Hatko [49], Wierzbicki/Kurcyusz [50], who use shifted penalty methods in order to compute solutions of problems with function space end condition, and Williamson/Polak [51]). Remark 3.13.…”

Section: Thenmentioning

confidence: 99%

The Maximum Principle for Relaxed Hereditary Differential Systems with Function Space end Condition

Colonius¹

1982

SIAM J. Control Optim.

View full text Add to dashboard Cite

This paper contains a proof of the global pointwise maximum principle for relaxed hereditary differential systems with general function space end condition. First a multiplier theorem establishes the existence of Lagrange multipliers (I0, I), where 10 R+ and is in the dual of the Sobolev space Wn'[-r, 0]. Then can be identified with an element of W"'[-r, 0] provided that the optimal trajectory satisfies a *

show abstract

“…Yet, even though hybrid models allow us to mathematically describe complex systems, efficient and flexible numerical methods to analyze and control those systems are hard to develop due to inherent phenomena such as Zeno executions [5]. In this paper we present a formulation for the class of hybrid systems whose trajectories are continuous (i.e., hybrid systems without reset maps) based on the theory of relaxed controls [6], [7], [8], and we show that our formulation allows us to numerically solve hybrid optimal control problems.…”

Section: Introductionmentioning

confidence: 99%

Optimal Control of Hybrid Systems Using a Feedback Relaxed Control Formulation

Westenbroek,

Gonzalez

2015

Preprint

View full text Add to dashboard Cite

We present a numerically tractable formulation for computing the optimal control of the class of hybrid dynamical systems whose trajectories are continuous. Our formulation, an extension of existing relaxed-control techniques for switched dynamical systems, incorporates the domain information of each discrete mode as part of the constraints in the optimization problem. Moreover, our numerical results are consistent with phenomena that are particular to hybrid systems, such as the creation of sliding trajectories between discrete modes.

show abstract

Relaxed Controls and the Convergence of Optimal Control Algorithms

Cited by 38 publications

References 9 publications

Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

The Maximum Principle for Relaxed Hereditary Differential Systems with Function Space end Condition

Optimal Control of Hybrid Systems Using a Feedback Relaxed Control Formulation

Contact Info

Product

Resources

About