Convergence analysis of adaptive critic based optimal control

Abstract

Adaptive critic based neural networks have been found to be powerful tools in solving various optimal control problems. The adaptive critic approach consists of two neural networks which output the control values and the Lagrangian multipliers associated with optimal control. These networks are trained successively and when the outputs of the two networks are mutually consistent and satisfy the differential constraints, the controller network output produces optimal control. In this paper, we analyze the mechanics of convergence of the network solutions. We establish the necessary conditions for the network solutions to converge and show that the converged solution is optimal.

Theoretical and Numerical Analysis of Approximate Dynamic Programming with Approximation Errors

“…Proof : The proof follows from [23], since, iterations given by (20) and (21) are exact VIs. Considering Lemmas 1 and 2 the following theorem proves the boundedness of the elements of…”

“…respectively, subject to dynamics (1), considering recursive relations (20) and (21). The following lemma provides the sufficient conditions for their convergence to the respective optimal value functions.…”

“…Lemma 2: The exact value iterations given by Eqs. (20) and (21) converge to the optimal value functions of cost functions (22) and (23), respectively, if they are initialized by smooth functions V 0 (.) and V 0 (.)…”

“…The most popular algorithm for ADP is Value Iteration (VI), [3], [18]. The convergence of VI for problems subject to this study, i.e., problems with undiscounted cost functions and continuous state and action spaces, was analyzed in [19], [20] for linear systems and in [21], [9], [23] for nonlinear systems. A crucial assumption in the cited convergence proofs is perfect function approximation, i.e., neglecting function approximation errors.…”