Model-based reinforcement learning for approximate optimal regulation

Kamalapurkar, Rushikesh; Walters, Patrick; Dixon, Warren E.

doi:10.1016/j.automatica.2015.10.039

Cited by 169 publications

(19 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to [36], the technique developed in this result implements simulation of experience in a model-based RL scheme by using the system model to extrapolate the approximate BE to unexplored areas of the state space. In the following, the trajectories of the state and the weight estimates W c and W a , evaluated at time t starting from appropriate initial conditions are denoted by x (t), W c (t) and W a (t), respectively.…”

Section: Velocity Estimator Designmentioning

confidence: 99%

Model-based reinforcement learning for output-feedback optimal control of a class of nonlinear systems

Kamalapurkar

2019

2019 American Control Conference (ACC)

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Velocity Estimator Designmentioning

confidence: 99%

Model-based reinforcement learning for output-feedback optimal control of a class of nonlinear systems

Kamalapurkar

2019

2019 American Control Conference (ACC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Concurrent learning (CL) [29][30][31], claims to relax the PE condition by storing information-rich past data along the system trajectory and concurrently using it along with the instantaneous data in the estimator design. A rank condition on a matrix, formed out of stored data, is sufficient to guarantee parameter convergence in the CL-based methods.…”

Section: Analytical Comparison With Existing Literaturementioning

confidence: 99%

“…Recent works [28][29][30] on learning and data-driven control methods have shown promise in improving tracking performance by using stored input-output data along the system trajectory which carries sufficient information about the unknown parameters. Girish et al [29,31] proposed a novel approach, coined as concurrent learning (CL) based model reference adaptive control (MRAC), where information-rich past data is stored and concurrently used along with gradient based parameter update laws.…”

Section: Introductionmentioning

confidence: 99%

Composite Adaptive Control of Uncertain Euler‐Lagrange Systems with Parameter Convergence without PE Condition

Roy

Bhasin

Kar

2018

Asian Journal of Control

View full text Add to dashboard Cite

This work proposes a novel composite adaptive controller for uncertain Euler-Lagrange (EL) systems. The composite adaptive law is strategically designed to be proportional to the parameter estimation error in addition to the tracking error, leading to parameter convergence. Unlike conventional adaptive control laws which require the regressor function to be persistently exciting (PE) for parameter convergence, the proposed method guarantees parameter convergence from a milder initially exciting (IE) condition on the regressor. The IE condition is significantly less restrictive than PE, since it does not rely on the future values of the signal and that it can be verified online. The proposed adaptive controller ensures exponential convergence of the tracking and the parameter estimation errors to zero once the sufficient IE condition is met. Simulation results corroborate the efficacy of the proposed technique and also establishes it's robustness property in the presence of unmodeled bounded disturbance.

show abstract

“…for all Z p such that Z p ∈ χ p and Z p ≥ v −1 lp (ι p ). Using the bounds in (40), the sufficient conditions in (41)- (43), and the inequality in (46), Theorem 4.18 in [47] can be invoked to conclude that every trajectory…”

Section: Stability Analysismentioning

confidence: 99%

Model-Based Reinforcement Learning in Differential Graphical Games

Kamalapurkar

Klotz

Walters

et al. 2018

IEEE Trans. Control Netw. Syst.

Self Cite

View full text Add to dashboard Cite

This paper seeks to combine differential game theory with the actor-critic-identifier architecture to determine forward-in-time, approximate optimal controllers for formation tracking in multi-agent systems, where the agents have uncertain heterogeneous nonlinear dynamics. A continuous control strategy is proposed, using communication feedback from extended neighbors on a communication topology that has a spanning tree. A model-based reinforcement learning technique is developed to cooperatively control a group of agents to track a trajectory in a desired formation. Simulation results are presented to demonstrate the performance of the developed technique.Rushikesh Kamalapurkar, is with the School system for each agent is a complex nonautonomous dynamical system. Nonautonomous systems, in general, have nonstationary value functions. Since non-stationary functions are difficult to approximate using parameterized function approximation schemes such as neural networks (NNs), designing optimal policies for nonautonomous systems is challenging.Since the external influence from neighbors renders the dynamics of each agent nonautonomous, optimization in a network of agents presents challenges similar to optimal tracking problems. Using insights gained from the authors' previous work on optimal tracking problems [39], this paper develops a model-based RL technique to generate feedback-Nash equilibrium policies online, for agents in a network with cooperative or competitive objectives. In particular, the network of agents is separated into autonomous subgraphs, and the differential game is solved separately on each subgraph.The primary contribution of this paper is the formulation and online approximate feedback-Nash equilibrium solution of an optimal network formation tracking problem. A relative control error minimization technique is introduced to facilitate the formulation of a feasible infinite-horizon total-cost differential graphical game. Dynamic programming-based feedback-Nash equilibrium solution of the differential graphical game is facilitated via the development of a set of coupled Hamilton-Jacobi (HJ) equations. The developed approximate feedback-Nash equilibrium solution is analyzed using a Lyapunov-based stability analysis to demonstrate ultimately bounded formation tracking in the presence of uncertainties.Patrick Walters received the Ph.D. degree in mechanical engineering from the University of Florida, Gainesville, FL, USA, in 2015. His research interests include reinforcement learning-based feedback control, approximate dynamic programming, and robust control of uncertain nonlinear systems with a focus on the application of underwater vehicles.Prof. Warren E.

show abstract

Model-based reinforcement learning for approximate optimal regulation

Cited by 169 publications

References 36 publications

Model-based reinforcement learning for output-feedback optimal control of a class of nonlinear systems

Model-based reinforcement learning for output-feedback optimal control of a class of nonlinear systems

Composite Adaptive Control of Uncertain Euler‐Lagrange Systems with Parameter Convergence without PE Condition

Model-Based Reinforcement Learning in Differential Graphical Games

Contact Info

Product

Resources

About