Value Iteration Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems

Abstract

In this paper, a value iteration adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon undiscounted optimal control problems for discrete-time nonlinear systems. The present value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize the algorithm. A novel convergence analysis is developed to guarantee that the iterative value function converges to the optimal performance index function. Initialized by different initial functions, it is proven that the iterative value function will be monotonically nonincreasing, monotonically nondecreasing, or nonmonotonic and will converge to the optimum. In this paper, for the first time, the admissibility properties of the iterative control laws are developed for value iteration algorithms. It is emphasized that new termination criteria are established to guarantee the effectiveness of the iterative control laws. Neural networks are used to approximate the iterative value function and compute the iterative control law, respectively, for facilitating the implementation of the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

Observer‐based fuzzy adaptive optimal control for nonlinear continuous‐time saturation interconnected systems

“…Consider the nonlinear continuous-time large system (2) with Assumptions 1 to 3 holding. If the fuzzy state observer is Equation (9), the critic FLSs is Equation (25); let the consequent update laws of the fuzzy state observer and the critic FLSs be Equations (13) and (41), respectively, and the control input with saturation constraints is chosen as Equation (37), then the interconnected system states x(t), the estimate errorx(t) of the interconnected system, the consequent estimate errorW S of the fuzzy state observer, and the consequent estimate errorW c of the critic FLS are ensured to be uniformly ultimately bounded.…”

“…For a nonlinear system, it is extremely difficult to derive an analytical solution of the Hamilton-Jacobi-Bellman (HJB) equations. Some scholars have used neural networks to approximate the solution of the HJB equation iteratively based on reinforcement learning and approximate-dynamic-programming (ADP) technology (see related works [24][25][26][27][28][29][30] ). Wei et al 24,25 proposed the ADP-based optimal control schemes for discrete-time nonlinear systems.…”

A novel triggering condition of event‐triggered control based on heuristic dynamic programming for discrete‐time systems

“…Then, several names emerged, eg, approximate dynamic programming and asymptotic dynamic programming. Iterative methods are widely used in ADP to obtain solutions of the Bellman equation indirectly . Adaptive dynamic programming can be divided into many categories, such as heuristic dynamic programming (HDP) dual heuristic dynamic programming (DHP), action dependent DHP (also called Q‐learning), globalized DHP, and so on.…”

Model Free Adaptive Control for a Class of Nonlinear Systems Using Quantized Information

“…Modeling these processes is difficult and sometimes impossible. Recently, a few data driven control (DDC) methods have been developed, such as model free adaptive control (MFAC) [1][2][3][4][5], iterative learning control (ILC) [6][7][8][9], virtual reference feedback tuning (VRFT) [10,11], iterative feedback tuning (IFT) [12,13], and adaptive dynamic programming (ADP) [14,15]. Of these methods, MFAC is an effective DDC approach for general discrete time nonlinear systems with unknown dynamics.…”