Koopman Spectrum Nonlinear Regulator and Provably Efficient Online Learning

Abstract

Most modern reinforcement learning algorithms optimize a cumulative single-step cost along a trajectory. The optimized motions are often 'unnatural', representing, for example, behaviors with sudden accelerations that waste energy and lack predictability. In this work, we present a novel paradigm of controlling nonlinear systems via the minimization of the Koopman spectrum cost: a cost over the Koopman operator of the controlled dynamics. This induces a broader class of dynamical behaviors that evolve over stable manifolds such as nonlinear oscillators, closed loops, and smooth movements. We demonstrate that some dynamics realizations that are not possible with a cumulative cost are feasible in this paradigm. Moreover, we present a provably efficient online learning algorithm for our problem that enjoys a sub-linear regret bound under some structural assumptions.

Dynamic Structure Estimation from Bandit Feedback

“…Instead, we consider systems with sub-Gaussian observation noise while allowing the observation to be made as bandit feedback. Our work includes algorithm for linear system, and we note that there is a recent surge of interests in studying controls and system identifications for dynamical systems (where states and/or actions are continuous) from sample complexity perspectives (e.g., Kakade et al [2020], Ohnishi et al [2021], Mania et al [2020], , Curi et al [2020]).…”

“…System identification has been widely studied in the controls community [Ljung, 2010], and recently, provably correct methods for identifications and controls of dynamical systems have gained attentions in the machine learning community as well (cf. Tsiamis and Pappas [2019], Tsiamis et al [2020], Lale et al [2020], Lee [2022], Kakade et al [2020], Ohnishi et al [2021], Mania et al [2020], , Curi et al [2020]).…”

Data-driven End-to-end Learning of Pole Placement Control for Nonlinear Dynamics via Koopman Invariant Subspaces

“…To alleviate such problems, end-to-end learning of controllers based on reinforcement learning [24,27] has been used, where controllers are directly optimized without separated system identification. Although many model-free reinforcement learning methods have been proposed, they typically require many training data since they do not model the dynamics.…”