A Nearer Optimal and Faster Trained Value Iteration ADP for Discrete-Time Nonlinear Systems

Abstract

Adaptive dynamic programming (ADP) is generally implemented using three neural networks: model network, action network, and critic network. In the conventional works of the value iteration ADP, the model network is initialized randomly and trained by the backpropagation algorithm, whose results are easy to get trapped in a local minimum; both the critic network and action network are trained in each outer-loop, which is time-consuming. To approximate the optimal control policy more accurately and decrease the value iteration ADP training time, we propose a nearer optimal and faster trained value iteration ADP for discrete-time nonlinear systems in this study. First, before training the model network with a backpropagation algorithm, we use a global searching method, i.e., genetic algorithm, to evolve the weights and biases of the neural network for a few generations. Second, in the outer-loop training process, we propose a trigger mechanism to decide whether to train the action network or not, which can save much training time. Examples of both linear and nonlinear systems are induced to verify the superiority of the proposed method compared with the conventional value iteration ADP. The simulation results show that the proposed algorithm can provide a nearer optimal control policy and save more training time than the conventional value iteration ADP.INDEX TERMS ADP, value iteration, genetic algorithm, trigger mechanism.

Decentralized position–force zero-sum approximate optimal control for reconfigurable robots with unmodeled dynamic

“…Optimal control method has been proposed nearly 70 years ago (Bellman and Dreyfus, 2013; Pontryagin, 1959), and plenty of literatures are recorded (An et al, 2021; An et al, 2021; Wei et al, 2021). Adaptive dynamic programming (ADP) theory is a key direction to undertake approximate optimal control system issues of discrete-time (Hu et al, 2021; Luo et al, 2020; Zhu et al, 2020a), continuous-time (An et al, 2021; Shan et al, 2020; Wei et al, 2020), data driven (Gao et al, 2018; Li et al, 2020; Su et al, 2020), and furthermore robot systems with input/output constraints (An et al, 2020; Lu et al, 2020; Ren et al, 2019), external disturbance (An et al, 2019; Song and Lewis, 2020; Xia et al, 2020), actuator failures (Dai et al, 2018; Jiao et al, 2018; Ma et al, 2020), and so on. Kong et al (2021b) developed n -link manipulator approximate optimal law via saturation nonlinearity.…”