Learning Output Reference Model Tracking for Higher-Order Nonlinear Systems with Unknown Dynamics

Abstract

This work suggests a solution for the output reference model (ORM) tracking control problem, based on approximate dynamic programming. General nonlinear systems are included in a control system (CS) and subjected to state feedback. By linear ORM selection, indirect CS feedback linearization is obtained, leading to favorable linear behavior of the CS. The Value Iteration (VI) algorithm ensures model-free nonlinear state feedback controller learning, without relying on the process dynamics. From linear to nonlinear parameterizations, a reliable approximate VI implementation in continuous state-action spaces depends on several key parameters such as problem dimension, exploration of the state-action space, the state-transitions dataset size, and a suitable selection of the function approximators. Herein, we find that, given a transition sample dataset and a general linear parameterization of the Q-function, the ORM tracking performance obtained with an approximate VI scheme can reach the performance level of a more general implementation using neural networks (NNs). Although the NN-based implementation takes more time to learn due to its higher complexity (more parameters), it is less sensitive to exploration settings, number of transition samples, and to the selected hyper-parameters, hence it is recommending as the de facto practical implementation. Contributions of this work include the following: VI convergence is guaranteed under general function approximators; a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multivariable aerodynamic process; comparisons with an offline deep deterministic policy gradient solution; implementation details and further discussions on the obtained results.

Reinforcement Q-Learning Incorporated With Internal Model Method for Output Feedback Tracking Control of Unknown Linear Systems

“…Proof. By Lemma 1, we know that the Q-function (27) in input-output form is equivalent to the original Q-function (14). Due to the excitation noise, the actual control input to collect data isû k = u k + w k with w k being the probing noise signals.…”

“…Most of the existing studies rely on the available measurement of full state information, see [22], [27] and the references therein. However, it may not be feasible in practical implementations [28], and therefore, it is desirable to design output feedback (OPFB) learning controllers.…”

Policy-Gradient and Actor-Critic Based State Representation Learning for Safe Driving of Autonomous Vehicles

“…Obtaining an optimal policy in reasonable time, taking decisions and actions under large state-spaces using DRL have been applied to network access, wireless caching, cognitive spectrum sensing, and network security. Some of the more recent DRL applications include modeling multiple experience pools for UAV autonomous motion planning in complex unknown environments [ 9 ], learning output reference model tracking for higher-order nonlinear systems with unknown dynamics [ 10 ], and pick and place operations in logistics using a mobile manipulator controlled with DRL [ 11 ]. The DRL paradigm has been extended to domains such as autonomous vehicles and has opened new research avenues [ 12 ].…”

Fisheye-Based Smart Control System for Autonomous UAV Operation

“…Responding to the needs, there have been various studies and algorithms developed for autonomous flight systems. Especially, many ML-based (Machine-learning based) methods have been proposed for autonomous path finding [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. However, they are limited when applied to a large target area.…”

“…In addition, Radac and Lala [ 23 ] showed that VIN algorithm’s convergence is guaranteed under general function approximators with providing a case study for a low-order linear system in order to generalize the more complex ORM tracking validation on a real-world nonlinear multi-variable aerodynamic process. Considering these advantages of VIN over DQN- or DDPG-based algorithms, we choose a VIN method as a baseline path planning algorithm.…”