Approximate Information States for Worst-case Control of Uncertain Systems

Abstract

Safety-critical cyber-physical systems require control strategies whose worst-case performance is robust against adversarial disturbances and modeling uncertainties. In this paper, we present a framework for approximate control and learning in partially observed systems to minimize the worstcase discounted cost over an infinite time horizon. We model disturbances to the system as finite-valued uncertain variables with unknown probability distributions. For problems with known system dynamics, we construct a dynamic programming (DP) decomposition to compute the optimal control strategy. Our first contribution is to define information states that improve the computational tractability of this DP without loss of optimality. Then, we describe a simplification for a class of problems where the incurred cost is observable at each time instance. Our second contribution is defining an approximate information state that can be constructed or learned directly from observed data for problems with observable costs. We derive bounds on the performance loss of the resulting approximate control strategy and illustrate the effectiveness of our approach in partially observed decision-making problems with a numerical example.

Approximate Information States for Worst-Case Control and Learning in Uncertain Systems

“…We bound the worst-case approximation loss rather than the expected loss. Note that we reported preliminary results for terminal-cost control problems with finite feasible sets in [59]. This paper extends the preliminary work as follows: (1) we consider worst-case instantaneous cost problems which subsume terminal cost problems; (2) we allow all variables to take values in continuous spaces; and (3) we illustrate the application of our results to a reinforcement learning problem.…”

“…A general notion of an information state for non-stochastic terminal cost problems was presented in [59]. Information states have also been derived for mixed problems considering both stochastic and non-stochastic objectives in [60] and for robust stochastic formulations in [61], [62].…”