Theoretical and Numerical Analysis of Approximate Dynamic Programming with Approximation Errors

Abstract

This study is aimed at answering the famous question of how the approximation errors at each iteration of Approximate Dynamic Programming (ADP) affect the quality of the final results considering the fact that errors at each iteration affect the next iteration. To this goal, convergence of Value Iteration scheme of ADP for deterministic nonlinear optimal control problems with undiscounted cost functions is investigated while considering the errors existing in approximating respective functions. The boundedness of the results around the optimal solution is obtained based on quantities which are known in a general optimal control problem and assumptions which are verifiable. Moreover, since the presence of the approximation errors leads to the deviation of the results from optimality, sufficient conditions for stability of the system operated by the result obtained after a finite number of value iterations, along with an estimation of its region of attraction, are derived in terms of a calculable upper bound of the control approximation error. Finally, the process of implementation of the method on an orbital maneuver problem is investigated through which the assumptions made in the theoretical developments are verified and the sufficient conditions are applied for guaranteeing stability and near optimality.

Stability Analysis of Optimal Adaptive Control Using Value Iteration With Approximation Errors

“…Let the approximation error be denoted with −1 (·). In other words instead of the exact value function V 0 (·) given by (9), one initiates the iterations using the approximate value function…”

“…Proof : The proof (which resembles Relaxed Dynamic Programming presented in [4]) provided in [9] applies to the case of initiating AVI using an admissible policy as well.…”

“…What makes their presence potentially problematic is the fact that the errors propagate throughout the iterations. Hence, regardless of how small they are, their accumulated effect can become significant, leading to a phenomenon similar to resonance, and hence, unreliability of the result, [9].…”

“…Analyzing VI under the presence of approximation errors, i.e. approximate VI (AVI), is an open research problem with a few published results, including [9]- [13]. Refs.…”

“…For example, the approximation error between the exact and approximate functions must be possible to be written in a multiplicative form in [13], instead of an additive form, as done in this study. As for [9], it investigated AVI with arbitrary initial guesses and after the training stage, i.e., it extended the result in [7] to AVI. The current study, however, investigates AVI initiated using a stabilizing guess and during the training phase.…”