Learning Optimal Controllers for Linear Systems With Multiplicative Noise via Policy Gradient

Abstract

The linear quadratic regulator (LQR) problem has reemerged as an important theoretical benchmark for reinforcement learning-based control of complex dynamical systems with continuous state and action spaces. In contrast with nearly all recent work in this area, we consider multiplicative noise models, which are increasingly relevant because they explicitly incorporate inherent uncertainty and variation in the system dynamics and thereby improve robustness properties of the controller. Robustness is a critical and poorly understood issue in reinforcement learning; existing methods which do not account for uncertainty can converge to fragile policies or fail to converge at all. Additionally, intentional injection of multiplicative noise into learning algorithms can enhance robustness of policies, as observed in ad hoc work on domain randomization. Although policy gradient algorithms require optimization of a nonconvex cost function, we show that the multiplicative noise LQR cost has a special property called gradient domination, which is exploited to prove global convergence of policy gradient algorithms to the globally optimum control policy with polynomial dependence on problem parameters. Results are provided both in the model-known and model-unknown settings where samples of system trajectories are used to estimate policy gradients

Online Actuator Selection and Controller Design for Linear Quadratic Regulation with Unknown System Model

“…The LQR problem with unknown system matrices (without the actuator or selection selection component) has been widely studied recently as a benchmark for reinforcement learning (e.g., [1,11,14,33,17,26]). The setting in this direction closest to ours is the so called model based learning, where the algorithms estimate the system matrices using the system trajectories and design the control policy based on the estimated system matrices.…”

Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

“…There has been some work on the case of noisy dynamics, but all in the setting of infinite horizon. In [26] the problem with a multiplicative noise was discussed, using a relatively straightforward extension of the deterministic dynamics considered in the Downloaded 10/18/21 to 88.109.64.71 Redistribution subject to SIAM license or copyright; see https://epubs.siam.org/page/terms original framework. In the case of additive noise, [32] studies the global convergence of policy gradient and other learning algorithms for the LQR over an infinite time horizon and with Gaussian noise.…”

On the Sample Complexity of Decentralized Linear Quadratic Regulator with Partially Nested Information Structure

“…One of the most studied problems is the centralized LQR problem. For this problem, two broad classes of methods have been studied, i.e., model based learning [1,29,12], and model-free learning [16,40,28,20]. In the model-based learning approach, a system model is first estimated from observed system trajectories using some system identification method.…”