Fitted Q-Function Control Methodology Based on Takagi–Sugeno Systems

Díaz, Henry; Armesto, Leopoldo; Sala, Antonio

doi:10.1109/tcst.2018.2885689

Cited by 9 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This work shows a successful example of a model-free output feedback controller used to collect input-to-state transition samples from the process for learning state-feedback ADP-based ORM tracking control. Therefore it fits with the recent data-driven control [35][36][37][38][39][40][41][42][43] and reinforcement learning [12,44,45] applications.…”

Section: Introductionsupporting

confidence: 56%

“…J ∞ MR from ( 5) obtained for γ = 1. The controller (36) will then close the feedback control loop as in…”

Section: Initial Controller With Model-free Vrftmentioning

confidence: 99%

“…ORM tracking is intended by making the closed loop CS match the same ORM M(z) = diag(M 1 (z), M 2 (z)). With the linear controller (36) used in closed-loop to stabilize the process, input-state-output data is collected for 7000 s. The reference inputs with amplitudes r k,1 ∈ [−2; 2], r k,2 ∈ [−1.4; 1.1] model successive steps that switch their amplitudes uniformly random at 17 s and 25 s, respectively. On the outputs u k,1 , u k,2 of both controllers C 11 (z), C 22 (z), an additive noise is added at every 2nd sample as an uniform random number in [−1.6; 1.6] for C 11 (z) and in [−1.7; 1.7] for C 22 (z).…”

Section: Input-state-output Data Collectionmentioning

confidence: 99%

See 2 more Smart Citations

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

Rădac

Lala

2019

Algorithms

View full text Add to dashboard Cite

This work suggests a solution for the output reference model (ORM) tracking control problem, based on approximate dynamic programming. General nonlinear systems are included in a control system (CS) and subjected to state feedback. By linear ORM selection, indirect CS feedback linearization is obtained, leading to favorable linear behavior of the CS. The Value Iteration (VI) algorithm ensures model-free nonlinear state feedback controller learning, without relying on the process dynamics. From linear to nonlinear parameterizations, a reliable approximate VI implementation in continuous state-action spaces depends on several key parameters such as problem dimension, exploration of the state-action space, the state-transitions dataset size, and a suitable selection of the function approximators. Herein, we find that, given a transition sample dataset and a general linear parameterization of the Q-function, the ORM tracking performance obtained with an approximate VI scheme can reach the performance level of a more general implementation using neural networks (NNs). Although the NN-based implementation takes more time to learn due to its higher complexity (more parameters), it is less sensitive to exploration settings, number of transition samples, and to the selected hyper-parameters, hence it is recommending as the de facto practical implementation. Contributions of this work include the following: VI convergence is guaranteed under general function approximators; a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multivariable aerodynamic process; comparisons with an offline deep deterministic policy gradient solution; implementation details and further discussions on the obtained results.

show abstract

Section: Introductionsupporting

confidence: 56%

“…J ∞ MR from ( 5) obtained for γ = 1. The controller (36) will then close the feedback control loop as in…”

Section: Initial Controller With Model-free Vrftmentioning

confidence: 99%

Section: Input-state-output Data Collectionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

Rădac

Lala

2019

Algorithms

View full text Add to dashboard Cite

show abstract

“…However, such methods open up the possibility of getting caught on local minima if they are not properly initialised. In our other work (Díaz et al, 2020), we propose LMI-based solutions as a starting point for Bellman error approaches.…”

Section: Introductionmentioning

confidence: 99%

A linear programming methodology for approximate dynamic programming

Díaz¹,

Sala²,

Armesto³

2020

Self Cite

View full text Add to dashboard Cite

The linear programming (LP) approach to solve the Bellman equation in dynamic programming is a well-known option for finite state and input spaces to obtain an exact solution. However, with function approximation or continuous state spaces, refinements are necessary. This paper presents a methodology to make approximate dynamic programming via LP work in practical control applications with continuous state and input spaces. There are some guidelines on data and regressor choices needed to obtain meaningful and well-conditioned value function estimates. The work discusses the introduction of terminal ingredients and computation of lower and upper bounds of the value function. An experimental inverted-pendulum application will be used to illustrate the proposal and carry out a suitable comparative analysis with alternative options in the literature.

show abstract

“…El problema fundamental de la programación dinámica aproximada es que las garantías de optimalidad se pierden al usar aproximadores V(x, θ) y, asimismo, los algoritmos iterativos clásicos PI o VI podrían no converger (Fairbank and Alonso, 2012), por pérdida de contractividad (Busoniu et al, 2010); ello podría requerir cambios en ellos, utilizando, por ejemplo, iteraciones descendiendo por gradiente en lo que se denomina minimización del residuo de Bellman (Antos et al, 2008;Díaz et al, 2018). Como alternativas a las metodologías iterativas, cambiando igualdades por desigualdades en la ecuación de Bellman, reformulaciones del problema como un problema de programación lineal pueden obtener, sin iteraciones, la soluciónóptima en casos de espacios de estado y control discretos, o una aproximación a la misma (De Farias and Van Roy, 2003).…”

Section: Introductionunclassified

Metodología de programación dinámica aproximada para control óptimo basada en datos

Díaz

Armesto

Sala

2019

Rev. iberoam. autom. inform. ind.

Self Cite

View full text Add to dashboard Cite

<p>En este artículo se presenta una metodología para el aprendizaje de controladores óptimos basados en datos, en el contexto de la programación dinámica aproximada. Existen soluciones previas en programación dinámica que utilizan programación lineal en espacios de estado discretos, pero que no se pueden aplicar directamente a espacios continuos. El objetivo de la metodología es calcular controladores óptimos para espacios de estados continuos, basados en datos, obtenidos mediante una estimación inferior del coste acumulado a través de aproximadores funcionales con parametrización lineal. Esto se resuelve de forma no iterativa con programación lineal, pero requiere proporcionar las condiciones adecuadas de regularización de regresores e introducir un coste de abandono de la región con datos válidos, con el fin de obtener resultados satisfactorios (evitando soluciones no acotadas o mal condicionadas).</p>

show abstract

Fitted Q-Function Control Methodology Based on Takagi–Sugeno Systems

Cited by 9 publications

References 40 publications

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

A linear programming methodology for approximate dynamic programming

Metodología de programación dinámica aproximada para control óptimo basada en datos

Contact Info

Product

Resources

About