Finite-time Identification of Stable Linear Systems Optimality of the Least-Squares Estimator

Abstract

We study contextual bandits with low-rank structure where, in each round, if the (context, arm) pair (i, j) ∈ [m] × [n] is selected, the learner observes a noisy sample of the (i, j)-th entry of an unknown low-rank reward matrix. Successive contexts are generated randomly in an i.i.d. manner and are revealed to the learner. For such bandits, we present efficient algorithms for policy evaluation, best policy identification and regret minimization. For policy evaluation and best policy identification, we show that our algorithms are nearly minimax optimal. For instance, the number of samples required to return an ε-optimal policy with probability at least 1 − δ typically scales as m+n ε 2 log(1/δ). Our regret minimization algorithm enjoys minimax guarantees scaling as r 7/4 (m + n) 3/4 √ T , which improves over existing algorithms. All the proposed algorithms consist of two phases: they first leverage spectral methods to estimate the left and right singular subspaces of the low-rank reward matrix. We show that these estimates enjoy tight error guarantees in the two-to-infinity norm. This in turn allows us to reformulate our problems as a misspecified linear bandit problem with dimension roughly r(m + n) and misspecification controlled by the subspace recovery error, as well as to design the second phase of our algorithms efficiently.

Finite-sample analysis of identification of switched linear systems with arbitrary or restricted switching

“…Furthermore, when the nominal system is asymptotically stable, i.e., a max < 1, we have λ max (λ T ) m/(1 − a 2 max ) from Corollary 4.1. Combing this bound with (8) shows that the estimation error of each mode is O 1/ |T i | , which matches the optimal decay rate of the LS estimator for asymptotically stable linear systems [4].…”

“…Note that the derived bounds for the Gramian in the following sections are also applicable to other finite-sample bounds for linear system identification [2], [4], [5], when extended to switched systems with deterministic switching, as the Gramian is an essential object in these bounds.…”

“…When the estimated model is used for controller design, the obtained error bound is important in understanding the effect of the estimation error on the control performance [1]. While several works consider the finite-sample analysis of linear system identification [2], [3], [4], [5], [6], [7], the finitesample analysis for identifying hybrid systems has rarely been addressed [8]. In this work, we consider the identification of a particular class of hybrid systems, i.e., switched linear systems (which consist of multiple linear systems corresponding to different modes), from the measured states and the measured switching signal.…”

Single Trajectory Nonparametric Learning of Nonlinear Dynamics

“…By contrast, previous work assumes i.i.d. data or focuses on either linear models [Simchowitz et al, 2018, Tsiamis and Pappas, 2019, Jedra and Proutiere, 2020 or parametric models with known nonlinearities , Sattar and Oymak, 2020, Jain et al, 2021.…”

“…Estimation of models (1) and (2) remains relatively poorly understood when the data is not i.i.d., with existing results being limited to when the function f is known to belong to certain parametric classes. In terms of parameter recovery, the LSE converges at a rate of T −1/2 for stable linear autoregressive systems f (x t ) = A x t [Simchowitz et al, 2018, Sarkar and Rakhlin, 2019, Jedra and Proutiere, 2020. The same rate can also be achieved for linear systems with more general input-output behavior Ozay, 2019, Tsiamis andPappas, 2019].…”