On adaptive Linear–Quadratic regulators

On a Phase Transition of Regret in Linear Quadratic Control: The Memoryless Case

“…On the theoretical side, the method of analysis to provide lower bounds here is novel and rests on comparing the Fisher information of any policy to the Fisher information of the optimal policy having knowledge of the system's parameters using singular subspace perturbation theory. This clarifies some connections between regret and parameter estimation ( [11], [13]) via the spectral properties of the Fisher information. We believe this proof strategy to potentially be of general interest.…”

“…In the uninformative regime, there is an asymptotically nonneglible trade-off between exploration and exploitation. These results partially answer an unresolved question in the literature [11], [12], as to precisely when logarithmic rates are attainable Ingvar Ziemann (ziemann@kth.se) and Henrik Sandberg (hsan@kth.se) are with the School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, SE-100 44 Stockholm, Sweden.…”

“…It is also interesting to note that the lower bound strongly suggests that a certainty equivalent controller perturbed by noise with full rank covariance of order 1/ √ t is a good idea since this corresponds closely to the case for which the lower bound is optimized to Ω( √ T ). Indeed, this was the strategy pursued by [11] attaining regret of order √ T for the full LQR. We also wish to mention that the concept of α-fast convergence is much inspired by the notion of uniformly fast convergence for the related bandit problems, see [20] and [21].…”

“…On the one hand, such extension could provide a Fisher information point of view on the LQR lower bound of [14]. On the other hand, this could help describe necessary and sufficient side information for logarithmic regret, [11]. Such situations can be covered by our method by adapting the parametrization of the problem which would result in a different (constrained) Fisher information.…”

“…for γ > 0, G 2 = GO 2 where G = ∇ B vec K(B, λ), I * t,B is given in (5), and O 2 is defined through (11).…”