Actor-Critic Reinforcement Learning for Control with Stability Guarantee

Han, Minghao; Zhang, Lixian; Wang, Jun; Pan, Wei

doi:10.48550/arxiv.2004.14288

Cited by 7 publications

(10 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Motivated by the works in [20], [25], [26], [27], we propose to incorporate the theoretical result in Theorem 1 to formulate a constrained optimisation problem, based on SAC [17]. First of all, a Lyapunov candidate needs to be selected at the first instance.…”

Section: Lyapunov-based Reinforcement Learningmentioning

confidence: 99%

“…[19] proposes a straightforward approach to construct the Lyapunov functions for nonlinear systems using DNNs. Recently, the asymptotic stability in model-free RL is given for robotic control tasks in [20]. Inspired by the works [19], [20], we will also parametrise the Lyapunov function as a DNN and learn the parameters from samples.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, the asymptotic stability in model-free RL is given for robotic control tasks in [20]. Inspired by the works [19], [20], we will also parametrise the Lyapunov function as a DNN and learn the parameters from samples. Thereafter, a new DRL algorithm based on soft actor-critic algorithm [17] that incorporates the Lyapunov boundedness condition in the objective function to be optimised is proposed.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

Hu¹,

Tang²,

Zhou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a deep reinforcement learning (DRL) algorithm for orientation estimation using inertial sensors combined with magnetometer. The Lyapunov's method in control theory is employed to prove the convergence of orientation estimation errors. Based on the theoretical results, the estimator gains and a Lyapunov function are parametrised by deep neural networks and learned from samples. The DRL estimator is compared with three well-known orientation estimation methods on both numerical simulations and real dataset collected from commercially available sensors. The results show that the proposed algorithm is superior for arbitrary estimation initialisation and can adapt to very large angular velocities for which other algorithms can be hardly applicable. To the best of our knowledge, this is the first DRL-based orientation estimation method with estimation error boundedness guarantee.

show abstract

Section: Lyapunov-based Reinforcement Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

Hu¹,

Tang²,

Zhou³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Analysis based on the Lyapunov theorem needs to construct a Lyapunov function to ensure the stability [10]. The Lyapunov stability methods have been widely used in the field of control engineering [10] and recently well investigated for RL algorithms to ensure the stability [11]. As another method, the input-output stability is also used to achieve the same purpose [12].…”

Section: Introductionmentioning

confidence: 99%

Stability-Certified Reinforcement Learning via Spectral Normalization

Takase¹,

Yoshikawa²,

Mariyama³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this article, two types of methods from different perspectives based on spectral normalization are described for ensuring the stability of the system controlled by a neural network. The first one is that the L2 gain of the feedback system is bounded less than 1 to satisfy the stability condition derived from the small-gain theorem. While explicitly including the stability condition, the first method may provide an insufficient performance on the neural network controller due to its strict stability condition. To overcome this difficulty, the second one is proposed, which improves the performance while ensuring the local stability with a larger region of attraction. In the second method, the stability is ensured by solving linear matrix inequalities after training the neural network controller. The spectral normalization proposed in this article improves the feasibility of the a-posteriori stability test by constructing tighter local sectors. The numerical experiments show that the second method provides enough performance compared with the first one while ensuring enough stability compared with the existing reinforcement learning algorithms.

show abstract

“…However, the classical control methods rely on the full or partial knowledge of the system dynamics to design controllers and are largely limited to systems with simple dynamics. Thus, it is a natural move to combine RL with control theory to develop learning control methods with a stability guarantee [5,6]. This paper was not presented at any IFAC meeting.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Han

Tian

Zhang

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Reinforcement learning (RL) is promising for complicated stochastic nonlinear control problems. Without using a mathematical model, an optimal controller can be learned from data evaluated by certain performance criteria through trial-and-error. However, the data-based learning approach is notorious for not guaranteeing stability, which is the most fundamental property for any control system. In this paper, the classic Lyapunov's method is explored to analyze the uniformly ultimate boundedness stability (UUB) solely based on data without using a mathematical model. It is further shown how RL with UUB guarantee can be applied to control dynamic systems with safety constraints. Based on the theoretical results, both off-policy and on-policy learning algorithms are proposed respectively. As a result, optimal controllers can be learned to guarantee UUB of the closed-loop system both at convergence and during learning. The proposed algorithms are evaluated on a series of robotic continuous control tasks with safety constraints. In comparison with the existing RL algorithms, the proposed method can achieve superior performance in terms of maintaining safety. As a qualitative evaluation of stability, our method shows impressive resilience even in the presence of external disturbances.

show abstract

Actor-Critic Reinforcement Learning for Control with Stability Guarantee

Cited by 7 publications

References 26 publications

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

Stability-Certified Reinforcement Learning via Spectral Normalization

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Contact Info

Product

Resources

About