Anticipating the long-term effect of online learning in control

Abstract

Control tasks with high levels of uncertainty and safety requirements are increasingly common. Typically, techniques that guarantee safety during learning and control utilize constraint-based safety certificates, which can be leveraged to compute safe control inputs. However, if model uncertainty is very high, the corresponding certificates are potentially invalid, meaning no control input satisfies the constraints imposed by the safety certificate. This paper considers a learning-based setting with a safety certificate based on a control barrier function second-order cone program. If the control barrier function certificate is valid, our approach leverages the control barrier function to guarantee safety. Otherwise, our method explores the system to recover the feasibility of the control barrier function constraint as fast as possible. To this end, we employ a method inspired by well-established tools from Bayesian optimization. We show that if the sampling frequency is high enough, we recover the feasibility of a control barrier function second-order cone program, guaranteeing safety. To the best of our knowledge, this corresponds to the first algorithm that guarantees safety through online learning without requiring a prior model or backup safe non-learning-based controller.

Learning against uncertainty in control engineering

“…where the expectation in (19) and the probability in (20) refer to the realization of the uncertainties/disturbance vector profile w. The condensed expression g(u, x k , w) ≤ 0 refers to the vector of constraints to be satisfied. This might gather stage constraints over the prediction horizon as well as terminal constraints at the end of the prediction horizon.…”

“…This is clearly shown on Figure 1 through the notation u 0 , u 1 , u 2 and u 3 . Note that the expression ( 26) is supposed to be an approximation of the expected cost as defined by (19). Similar weighted constraints can be similarly defined in order to replace (20).…”

“…Among the interesting topics that are not mentioned in this paper, it is worth mentioning those related to the construction of non parametric models (kernel-based [42], kinky inference-based [18,43] among many others) and the approximation of associated bounds (maximum modeling error, Lypschitz constants to cite but few ones) that allow the use of standard control design or proof arguments that require the knowledge of such bounds. Recent works related to the propagation of uncertainties [65], anticipating the learning process [19], joint extended state/parameters estimation error certification [7], combining system identification and reinforcement learning [44] or MPC-induced parametrization of reinforcement learning [28] are worth examining and might be very promising topics in the near future.…”

Confidence Regions for Simulations with Learned Probabilistic Models

“…Therefore, we can iteratively sample F x k from the conditional probabilities P Fx k |Fx k−1 ,...,Fx 0 , such that the samples satisfy (7). Furthermore, we can pursue the same procedure to obtain samples for the observation model {H(x, ψ)} x∈X due to the equivalence of both sampling problems.…”

“…For example, the consideration of the uncertainty of learned models leads to cautiousness in model predictive control [4], and can be efficiently implemented using scenario approaches [5]. Furthermore, uncertainty awareness in model-based reinforcement learning automatically balances system identification and task execution [6], and allows to anticipate the effect of online learning during policy optimization [7].…”