Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Abstract

Gradient-based methods have been widely used for system design and optimization in diverse application domains. Recently, there has been a renewed interest in studying theoretical properties of these methods in the context of control and reinforcement learning. This article surveys some of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that has been popularized by successes of reinforcement learning. We take an interdisciplinary perspective in our exposition that connects control theory, reinforcement learning, and large-scale optimization. We review a number of recently developed theoretical results on the optimization landscape, global convergence, and sample complexityof gradient-based methods for various continuous control problems, such as the linear quadratic regulator (LQR), [Formula: see text] control, risk-sensitive control, linear quadratic Gaussian (LQG) control, and output feedback synthesis. In conjunction with these optimization results, we also discuss how direct policy optimization handles stability and robustness concerns in learning-based control, two main desiderata in control engineering. We conclude the survey by pointing out several challenges and opportunities at the intersection of learning and control.

How to Design Reinforcement Learning Methods for the Edge: An Integrated Approach toward Intelligent Decision Making

“…Due to the above reasons, reinforcement learning based on value optimization cannot be adapted to all scenarios, so the methods based on policy optimization (PO) are born [111]. Policy optimization reinforcement learning methods are usually divided by the determinism of the policy, where the deterministic policy approach will only choose a fixed action at each step and then enter the next state; furthermore, the policy of the stochastic policy approach is a probability distribution of an action (one out of several actions), and the agent will choose one according to the current state with a certain policy and then enter the next state.…”

AI Institute in Dynamic Systems: Developing machine learning and AI tools for scientific discovery, engineering design, and data‐driven control

“…Thrust 1 enables Thrust 2, which aims to build a diversity of dynamic models directly from sensor data (Chen et al 2022;Fasel et al 2022;Gao and Kutz 2022). With the development of dynamic models, datadriven control protocols are developed in Thrust 3 (Hu et al 2023). The understanding of models and controls then re-integrates with sensing where better sensing strategies can be constructed with knowledge from Thrusts 2 and 3.…”

“…This requires new perspectives and computational schemes to address the emerging challenges posed for optimization in the context of complex dynamical systems. Thus, the institute has leveraged the structure of dynamic data to design guaranteed optimization and sensing strategies for safe, efficient, robust real-time decision making (Hu et al 2023;Manohar et al 2018).…”

“…Thus, the institute is innovating algorithms for real-time control of dynamic systems for their safe, reliable, and efficient operation leveraging principles from model-based and model-free control, optimization, and learning methods developed across all the three thrusts of modeling, control, and sensing/optimization. Institute partners are already establishing the theoretical foundations of policy optimization for learning control policies, thus revolutionizing how modern RL control can be embedded in engineering systems (Hu et al 2023).…”

“…As such, the AI Institute develops the fundamental mathematical and computational tools required to empower AI for real-time sensing (Manohar et al 2018), learning (Chen et al 2022;Fasel et al 2022), decision making (Hu et al 2023), and prediction capabilities that lead the way towards safe, reliable, efficient, explainable, and ethical data-enabled engineering and science systems. The institute is organized around three key thrusts that are used in synergistic partnership for solving today's grandchallenge problems.…”