Robust control under worst‐case uncertainty for unknown nonlinear systems using modified reinforcement learning

Abstract

Reinforcement learning (RL) is an effective method for the design of robust controllers of unknown nonlinear systems. Normal RLs for robust control, such as actor-critic (AC) algorithms, depend on the estimation accuracy. Uncertainty in the worst case requires a large state-action space, this causes overestimation and computational problems. In this article, the RL method is modified with the k-nearest neighbor and the double Q-learning algorithm. The modified RL does not need the neural estimator as AC and can stabilize the unknown nonlinear system under the worst-case uncertainty. The convergence property of the proposed RL method is analyzed. The simulations and the experimental results show that our modified RLs are much more robust compared with the classic controllers, such as the proportional-integral-derivative, the sliding mode, and the optimal linear quadratic regulator controllers. K E Y W O R D Sk-nearest neighbors, double estimator, overestimation, robust reward, state-action space, worst-case uncertainty INTRODUCTIONThe objective of robust control is to achieve robust performance in presence of disturbances. Most robust controllers are inspired by the optimal control theory, such as  2 control, 1,2 which minimizes a certain cost function to find an optimal controller. The most popular  2 controller is the linear quadratic regulator (LQR). 3 It does not work well in presence of disturbances. The  ∞ control can find a robust controller when the system has disturbances. Its performance is poor compared with the  2 control. 4 The combination of  2 and  ∞ , called  2 ∕ ∞ control, has both advantages, that is., it has optimal performance with bounded disturbances. 5 The  2 ∕ ∞ controller design needs a complete knowledge of the system dynamics. 6 These controllers are model-based. The time-varying quadratic optimization can be calculated by the zeroing neural network. It can simultaneously achieve the finite-time convergence and inherent noise tolerance. 7 However, it requires prefect activation functions. Model-free controllers, like the proportional-integral-derivative (PID) control, 8,9 the sliding mode control (SMC), 10 neural control, 11-16 among others, do not require dynamic knowledge of the system. However, parameter tuning and some prior knowledge of the disturbances prevent these model-free controllers to perform optimally.Reinforcement learning (RL) 17,18 is another effective method without models. It is designed in the sense of  2 control. 19 The temporal difference (TD) rule, such as Q-learning, is applied to find an optimal solution for Markov decision processes. 20 The advantage of RL over the other model-free methods is that it can reach optimal performances. Recent results show that RL methods can learn  2 and  ∞ controllers without system dynamics. 21 2920The main objective of RL for  2 and  ∞ is to minimize the total accumulative reward. For a robust controller, the reward is designed in the sense of control problems  2 and  ∞ . This robust reward can be static or dynam...

Human-Behavior Learning for Infinite-Horizon Optimal Tracking Problems of Robot Manipulators

“…The cognitive set C gives numerous models and skills that are useful to get the optimal decision making (13). Humanbehavior is supported by many intelligent techniques such as: reinforcement learning [8], [32], deep learning [17], [26], machine learning techniques [21], function approximators [5], [8], and so on; with the aim of providing an ability to learn by interacting with the actions (x t , u t ) to achieve the control task.…”

“…The neural cognitive model (21) gives an approximate solution of (11) such that the optimal decision making control ( 13) is obtained. To achieve this goal, the approximation needs an exploration term which, in this case, is given by a PE exciting condition.…”

“…This exploration is satisfied if q t fulfills a persistently exciting (PE) condition [31]. So, the PE skill is within the neural approximator (21), that is, q t = q t (x t , u 2,t ; P E).…”

“…The neural model (21) updates the weights by considering only one sample [19]. The key idea of the experience replay is to store more samples in a memory vector and add them to the gradient descent update rule [12], such that it minimizes simultaneously the actual TD error δ t and the TD error of previous samples.…”

A Novel Tuning Method of PD With Gravity Compensation Controller for Robot Manipulators

“…To satisfy the control objectives, it has been developed different control techniques, such as PID [7]- [9], sliding mode control (SMC) [10]- [12], neural networks [13], intelligent techniques [14], [15] or even linear controllers [2], [16]. Each algorithm is capable to compensate the gravitational term and robustify the control law.…”