Actor-Critic Reinforcement Learning for Control With Stability Guarantee

Han, Minghao; Zhang, Lixian; Wang, Jun; Pan, Wei

doi:10.1109/lra.2020.3011351

Cited by 83 publications

(50 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [28] multi pseudo Qlearning-based deterministic policy gradient algorithm was proposed to achieve high-level tracking control accuracy of AUVs, which validated that increasing the number of the actors and critics could further improve the performance. Recently, a data-based approach for analyzing the stability of discrete-time nonlinear stochastic systems modeled by Markov decision process, by using the classic Lyapunov's method in control theory [29]. Due to the limited exploration ability caused deterministic policy, high-speed autonomous drifting is addressed, using a closed-loop controller based on the deep RL algorithm soft actor critic (SAC) to control the steering angle and throttle of simulated vehicles in [30].…”

Section: Introduction a Related Workmentioning

confidence: 99%

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

Gao

Liang

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Tracking control is an essential capability for nonholonomic wheeled mobile robots (NWMR) to achieve autonomous navigation. This paper presents a novel hybrid control strategy combined modebased control and actor-critic based deep reinforcement learning method. Based on the Lyapunov method, a kinematics control law named given control is obtained with pose errors. Then, the tracking control problem is converted to a finite Markov decision process, in which the defined state contains current tracking errors, given control inputs and one-step errors. After training with deep deterministic policy gradient method, the action named acquired control inputs is capable of compensating the existing errors. Thus, the hybrid control strategy is obtained under velocity constraint, acceleration constraint and bounded uncertainty. A cumulative error is also defined as a criteria to evaluate tracking performance. The comparison results in simulation demonstrate that our proposed method have an obviously advantage on both tracking accuracy and training efficiency.INDEX TERMS Deep reinforcement learning, tracking control, kinematics control, hybrid control strategy, nonholonomic wheeled mobile robot.

show abstract

Section: Introduction a Related Workmentioning

confidence: 99%

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

Gao

Liang

et al. 2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…[19] proposes a straightforward approach to construct the Lyapunov functions for nonlinear systems using DNNs. Recently, the asymptotic stability in model-free RL is given for robotic control tasks in [20]. Inspired by the works [19], [20], we will also parametrise the Lyapunov function as a DNN and learn the parameters from samples.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, the asymptotic stability in model-free RL is given for robotic control tasks in [20]. Inspired by the works [19], [20], we will also parametrise the Lyapunov function as a DNN and learn the parameters from samples. Thereafter, a new DRL algorithm based on soft actor-critic algorithm [17] that incorporates the Lyapunov boundedness condition in the objective function to be optimised is proposed.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

Tang

Zhou

et al. 2021

2021 IEEE International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

This paper presents a deep reinforcement learning (DRL) algorithm for orientation estimation using inertial sensors combined with magnetometer. The Lyapunov's method in control theory is employed to prove the convergence of orientation estimation errors. Based on the theoretical results, the estimator gains and a Lyapunov function are parametrised by deep neural networks and learned from samples. The DRL estimator is compared with three well-known orientation estimation methods on both numerical simulations and real dataset collected from commercially available sensors. The results show that the proposed algorithm is superior for arbitrary estimation initialisation and can adapt to very large angular velocities for which other algorithms can be hardly applicable. To the best of our knowledge, this is the first DRL-based orientation estimation method with estimation error boundedness guarantee.

show abstract

“…It implies that the tracking error converges to zero on the sliding surface s L = 0, i.e. q L → q Lr and trueq̇L→trueq̇Lr as t →∞. For the follower robot, the infinite-horizon return function serves as Lyapunov function for the reinforcement learning system (Kamalapurkar et al , 2017; Han et al , 2020), such that: where γ ∈ [0,1] is the discounting factor. The reward function (13) is included in the Lyapunov function and the reward is larger than zero through setting the term weights, i.e.…”

mentioning

confidence: 99%

“…For the follower robot, the infinite-horizon return function serves as Lyapunov function for the reinforcement learning system (Kamalapurkar et al , 2017; Han et al , 2020), such that: …”

mentioning

confidence: 99%

A robotic system with reinforcement learning for lower extremity hemiparesis rehabilitation

Cheng

et al. 2021

View full text Add to dashboard Cite

Purpose This paper aims to propose a bilateral robotic system for lower extremity hemiparesis rehabilitation. The hemiplegic patients can complete rehabilitation exercise voluntarily with the assistance of the robot. The reinforcement learning is included in the robot control system, enhancing the muscle activation of the impaired limbs (ILs) efficiently with ensuring the patients’ safety. Design/methodology/approach A bilateral leader–follower robotic system is constructed for lower extremity hemiparesis rehabilitation, where the leader robot interacts with the healthy limb (HL) and the follow robot is worn by the IL. The therapeutic training is transferred from the HL to the IL with the assistance of the robot, and the IL follows the motion trajectory prescribed by the HL, which is called the mirror therapy. The model reference adaptive impedance control is used for the leader robot, and the reinforcement learning controller is designed for the follower robot. The reinforcement learning aims to increase the muscle activation of the IL and ensure that its motion can be mastered by the HL for safety. An asynchronous algorithm is designed by improving experience relay to run in parallel on multiple robotic platforms to reduce learning time. Findings Through clinical tests, the lower extremity hemiplegic patients can rehabilitate with high efficiency using the robotic system. Also, the proposed scheme outperforms other state-of-the-art methods in tracking performance, muscle activation, learning efficiency and rehabilitation efficacy. Originality/value Using the aimed robotic system, the lower extremity hemiplegic patients with different movement abilities can obtain better rehabilitation efficacy.

show abstract

Actor-Critic Reinforcement Learning for Control With Stability Guarantee

Cited by 83 publications

References 25 publications

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

A Hybrid Tracking Control Strategy for Nonholonomic Wheeled Mobile Robot Incorporating Deep Reinforcement Learning Approach

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

A robotic system with reinforcement learning for lower extremity hemiparesis rehabilitation

Contact Info

Product

Resources

About