Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

Abstract

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework, and that it is more robust with respect to model misspecification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.

Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

“…3) with optimal K t = K \ast t as given by (2.5). We have the following result on the well-definedness of P K K K t , the proof of which can be found in [29].…”

“…To utilize Lemmas 3.6 and 3.7 in the proof of Theorem 3.3, we need to further bound P t and \Sigma K K K , which is provided below in Lemma 3.8. The proof can be found in [29].…”

“…We defer the proof of Lemma 3.6 to [29]. Lemma 3.6 implies that when the gradient becomes small, the value of the objective function is close to C(K K K \ast ).…”

“…We provide some technical proofs in the longer arXiv version of our paper [29], which can be found at https://arxiv.org/pdf/2011.10300.pdf.…”

“…We defer the proof of Lemma 3.7 to [29]. To see why Lemma 3.7 is related to the smoothness, observe that when K K K \prime is sufficiently close to K K K, in the sense that…”