Regret Analysis of Learning-Based MPC with Partially-Unknown Cost Function

Abstract

The exploration/exploitation trade-off is an inherent challenge in data-driven and adaptive control. Though this trade-off has been studied for multi-armed bandits, reinforcement learning (RL) for finite Markov chains, and RL for linear control systems; it is less well-studied for learning-based control of nonlinear control systems. A significant theoretical challenge in the nonlinear setting is that, unlike the linear case, there is no explicit characterization of an optimal controller for a given set of cost and system parameters. We propose in this paper the use of a finite-horizon oracle controller with perfect knowledge of all system parameters as a reference for optimal control actions. First, this allows us to propose a new regret notion with respect to this oracle finite-horizon controller. Second, this allows us to develop learning-based policies that we prove achieve low regret (i.e., square-root regret up to a log-squared factor) with respect to this oracle finite-horizon controller. This policy is developed in the context of learning-based model predictive control (LBMPC). We conduct a statistical analysis to prove finite sample concentration bounds for the estimation step of our policy, and then we perform a control-theoretic analysis using techniques from MPC-and optimization-theory to show this policy ensures closed-loop stability and achieves low regret. We conclude with numerical experiments on a model of heating, ventilation, and air-conditioning (HVAC) systems that show the low regret of our policy in a setting where the cost function is partially-unknown to the controller.

Suboptimality analysis of receding horizon quadratic control with unknown linear systems and its applications in learning-based control

“…The approach achieves Õ(T 2/3 ) regret, where Õ(•) represents the order up to logarithmic factors. The authors in [28] consider a nonlinear MPC with constraints and state feedback, where the nonlinear dynamics is decoupled into a known nominal linear model and an additive unknown nonlinear term. The unknown nonlinear term is re-estimated during the closed-loop operation, and to ensure the data informativity for estimation, at each time step, the control algorithm randomly chooses to either inject a random input within the input constraint or applies the input computed by the MPC controller.…”

“…The results in [26]- [28] have made significant contributions in designing learning-based RHC controllers for unknown systems and in understanding their finite-time performances. However, there are still important open problems.…”

“…However, there are still important open problems. While the prediction horizon of an RHC controller is an important design parameter, it is chosen to be a fixed value in [26]- [28]. The impact of prediction horizon on the control performance is not explicitly investigated in their finite-time performance analysis.…”

“…The impact of prediction horizon on the control performance is not explicitly investigated in their finite-time performance analysis. In addition, when the system is known, it is well known in reinforcement learning [29] that a more accurate terminal value function that is closer to the optimal value function can improve the control performance; however, this effect of terminal value function is not investigated in [26]- [28] for the setting with an unknown system.…”

Online convex optimization for constrained control of linear systems using a reference governor

“…Dynamic regret is able to capture transient performance of the closed loop system and is defined as the ⋆ This work was supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -505182457. difference between the accumulated closed-loop cost of the controller and some benchmark, which is typically defined in hindsight, i.e., with knowledge of all cost functions. Studying the dynamic regret of controllers for dynamical systems has recently gained increasing attention, compare, e.g., Dogan et al (2021); Didier et al (2022). However, in the literature on OCO-based control, pointwise in time state and input constraints are only considered in Nonhoff and Müller (2021); Li et al (2021), and restrictive assumptions or a limited setting are necessary in these works to guarantee constraint satisfaction at all times.…”