A Tutorial on Concentration Bounds for System Identification

Matni, Nikolai; Tu, Stephen

doi:10.1109/cdc40024.2019.9029621

Cited by 33 publications

(23 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This tutorial paper and our companion paper [7] presented a broad overview of recent progress towards the finitetime analysis for reinforcement learning and adaptive control methods. We have attempted to provide a summary of representative results in this space that establish connections between the adaptive control literature and methods recently proposed in reinforcement learning.…”

Section: Discussionmentioning

confidence: 99%

“…We combine the techniques described in [7] with robust and optimal control to derive finite-time guarantees for the optimal LQR control of an unknown system. We partition our study according to three initial uncertainty regimes: (i) completely unknown (A, B), (ii) moderate error bounds under which CE control may fail, and (iii) small error bounds under which CE control is stabilizing.…”

Section: Model-based Methods For Lqrmentioning

confidence: 99%

“…In this tutorial paper and our companion paper [7], we highlight recent advances that provide non-asymptotic analysis of adaptive algorithms. Our aim is for these papers is for them to serve as a jumping off point for control theorists wanting to work in RL problems.…”

Section: Introductionmentioning

confidence: 99%

“…Our aim is for these papers is for them to serve as a jumping off point for control theorists wanting to work in RL problems. In [7], we present an overview of tools and results on finite-data guarantees for system identification. This paper focuses on finite-data guarantees for self-tuning and adaptive control strategies, and is structured as follows: 4) Dynamic Programming and Reinforcement Learning: A major part of the literature on dynamic programming is devoted to "tabular MDPs," i.e.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

From self-tuning regulators to reinforcement learning and back again

Matni

Proutière

Rantzer

et al. 2019

2019 IEEE 58th Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

Machine and reinforcement learning (RL) are being applied to plan and control the behavior of autonomous systems interacting with the physical world -examples include self-driving vehicles, distributed sensor networks, and agile robots. However, if machine learning is to be applied in these new settings, the resulting algorithms must come with the reliability, robustness, and safety guarantees that are hallmarks of the control theory literature, as failures could be catastrophic. Thus, as RL algorithms are increasingly and more aggressively deployed in safety critical settings, it is imperative that control theorists be part of the conversation. The goal of this tutorial paper is to provide a jumping off point for control theorists wishing to work on RL related problems by covering recent advances in bridging learning and control theory, and by placing these results within the appropriate historical context of the system identification and adaptive control literatures.• Section II: provides an extensive literature review of work spanning classical and modern results in system identification, adaptive control, and RL.• Section III: introduces the fundamental problem and performance metrics considered in RL, and relates them to examples familiar to the controls community.• Section IV: provides a survey of contemporary results for problems with finite state and action spaces. • Section V: shows how system estimates and error bounds can be incorporated into model-based self-tuning regulators with finite-time performance guarantees.• Section VI: presents guarantees for model-free methods, and shows that a complexity gap exists between model-based and model-free methods. II. LITERATURE REVIEWThe results we present in this paper draw heavily from three broad areas of control and learning theory: system identification, adaptive control, and approximate dynamic programming (ADP) or, as it has come to be known, reinforcement learning. Each of these areas has a long and rich history and a general literature review is outside the scope of this tutorial. Below we will instead emphasize pointers to good textbooks and survey papers, before giving a more careful account of recent work.1) System Identification: The estimation of system behavior from input/output experiments has a well-developed theory dating back to the 1960s, particularly in the case of linear-time-invariant systems. Standard reference texts on the topic include [6], [8], [9], [10]. The success of discrete time series analysis by Box and Jenkins [11] provided an early impetus for the extension of these methods to the controlled system setting. Important connections to information theory were established by Akaike [12]. The rise of robust control in the 1980s further inspired system identification procedures, wherein model errors were optimized under the assumption of adversarial noise processes [13]. Another important step was the development of subspace methods [14], which became a powerful tool for identification of multi-input multi-output systems.2) Adaptive...

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Model-based Methods For Lqrmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

From self-tuning regulators to reinforcement learning and back again

Matni

Proutière

Rantzer

et al. 2019

2019 IEEE 58th Conference on Decision and Control (CDC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The connection of RL with optimal control is known since ever (see, e.g., [15,38,63]). According to [48,49] tools stemming from advanced control theory should enhance RL in general.…”

Section: Introductionmentioning

confidence: 99%

Machine Learning and Control Engineering: The Model-Free Case

Fliesś

Join

2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

This paper states that Model-Free Control (MFC), which must not be confused with Model-Free Reinforcement Learning, is a new tool for Machine Learning (ML). MFC is easy to implement and should be substituted in control engineering to ML via Artificial Neural Networks and/or Reinforcement Learning. A laboratory experiment, which was already investigated via today's ML techniques, is reported in order to confirm this viewpoint.

show abstract

Data‐driven control for networked systems with multiple packet dropouts

Chen

2023

Intl J Robust & Nonlinear

View full text Add to dashboard Cite

This paper investigates data-driven control for a class of networked control systems with multiple packet dropouts and unknown system parameters. Here, multiple packet dropouts occur randomly in the controller-to-actuator (C/A) and sensor-to-controller (S/C) channels, where successive packet dropouts are limited by known upper bounds. By introducing the summation inequality, a model-based mean-square asymptotic stability condition is established for the closed-loop networked control system with the potential to migrate to a data-driven solution. Then, the data-driven mean-square asymptotic stability condition for the closed-loop system is presented by merging the model-based stability condition with noisy data-parameterized representations. On this basis, the data-driven controller gain design approaches for combining unknown and known input gain matrices are presented in turn. Finally, two simulation examples are provided to show the applicability of the developed approaches.

show abstract

A Tutorial on Concentration Bounds for System Identification

Cited by 33 publications

References 14 publications

From self-tuning regulators to reinforcement learning and back again

From self-tuning regulators to reinforcement learning and back again

Machine Learning and Control Engineering: The Model-Free Case

Data‐driven control for networked systems with multiple packet dropouts

Contact Info

Product

Resources

About