Task-Optimal Exploration in Linear Dynamical Systems

Abstract

Exploration in unknown environments is a fundamental problem in reinforcement learning and control. In this work, we study task-guided exploration and determine what precisely an agent must learn about their environment in order to complete a particular task. Formally, we study a broad class of decision-making problems in the setting of linear dynamical systems, a class that includes the linear quadratic regulator problem. We provide instance-and taskdependent lower bounds which explicitly quantify the difficulty of completing a task of interest. Motivated by our lower bound, we propose a computationally efficient experiment-design based exploration algorithm. We show that it optimally explores the environment, collecting precisely the information needed to complete the task, and provide finite-time bounds guaranteeing that it achieves the instance-and task-optimal sample complexity, up to constant factors. Through several examples of the LQR problem, we show that performing task-guided exploration provably improves on exploration schemes which do not take into account the task of interest. Along the way, we establish that certainty equivalence decision making is instance-and task-optimal, and obtain the first algorithm for the linear quadratic regulator problem which is instance-optimal. We conclude with several experiments illustrating the effectiveness of our approach in practice.

Beyond No Regret: Instance-Dependent PAC Reinforcement Learning

“…We discuss (Marjani et al, 2021) in more detail in Section 4.1. In the special case of linear dynamical systems and smooth rewards, a setting which encompasses the Linear Quadratic Regulator problem, Wagenmaker et al (2021) establish a finite-time, instancedependent lower bound and matching upper bound for -optimal policy identification. To our knowledge, this is the only work to obtain an instance-optimal ( , δ)-PAC result, but their analysis does not apply to tabular MDPs.…”

The Fundamental Limitations of Learning Linear-Quadratic Regulators

“…Note that L provides an upper bound on the information in the face of an optimal offline exploration policy. Studying it may therefore assist with experiment design, as in Wagenmaker et al (2021). Rather than the appearance of −K I on the denominator, as we saw in Proposition 3.1, we have A B F .…”

“…Lower bounds on the variance of the gradient estimates in policy gradient approaches are supplied in Ziemann et al (2022). Lower bounds for offline linear control are also studied in Wagenmaker et al (2021) with the objective of designing optimal experiments. We instead focus on the LQR setting to understand the dependence of the excess cost on interpretable system-theoretic quantities.…”