From Dissipativity Theory to Compositional Construction of Finite Markov Decision Processes

Abstract

This paper is concerned with a compositional approach for constructing finite Markov decision processes of interconnected discrete-time stochastic control systems. The proposed approach leverages the interconnection topology and a notion of so-called stochastic storage functions describing joint dissipativitytype properties of subsystems and their abstractions. In the first part of the paper, we derive dissipativitytype compositional conditions for quantifying the error between the interconnection of stochastic control subsystems and that of their abstractions. In the second part of the paper, we propose an approach to construct finite Markov decision processes together with their corresponding stochastic storage functions for classes of discrete-time control systems satisfying some incremental passivablity property. Under this property, one can construct finite Markov decision processes by a suitable discretization of the input and state sets. Moreover, we show that for linear stochastic control systems, the aforementioned property can be readily checked by some matrix inequality. We apply our proposed results to the temperature regulation in a circular building by constructing compositionally a finite Markov decision process of a network containing 200 rooms in which the compositionality condition does not require any constraint on the number or gains of the subsystems. We employ the constructed finite Markov decision process as a substitute to synthesize policies regulating the temperature in each room for a bounded time horizon.

Compositional abstraction-based synthesis for networks of stochastic switched systems

“…We show that our proposed results here which are based on max small-gain conditions significantly outperform the results provided in [SAM17] and [LSZ18b] which are respectively based on dynamic Bayesian network (DBN) and dissipativity-type conditions. This outperformance is due to the fact that the approximation error in [SAM17] and [LSZ18b] increases as the number of subsystems grows. Whereas, our error provided in (3.5) does not change since the overall approximation error is completely independent of the size of the network, and is computed only based on the maximum error of subsystems instead of being a linear combination of them which is the case in [SAM17] and [LSZ18b].…”

“…(8.8)), but with the gain of providing an overall error for the network only based on the maximum error of subsystems instead of a linear combination of them. Thus, our proposed results here outperform the ones in [LSZ18b] for large-scale stochastic switched systems admitting a common Lyapunov function. 6.6.…”

“…By increasing the number of subsystems, the error in (3.5) does not change since the overall ψ is independent of N , and is computed only based on the maximum of ψ i of subsystems instead of being a linear combination of them which is the case in [LSZ18b]. Nevertheless, for networks with small number of subsystems, the proposed errors in [LSZ18b] are slightly better than the ones provided in this work. This issue is expected and the reason is due to the conservatism nature of the approach that we employ here ([SGZ18, Theorem 1]) to transfer the additive form of our pseudo-simulation functions to a max form (cf.…”

“…We apply our proposed techniques to a fully interconnected network of 500 nonlinear subsystems (totally 1000 dimensions) and construct their finite MDPs with guaranteed error bounds. We compare our results with the compositional techniques proposed in [SAM17] and [LSZ18b].…”

Compositional abstraction of large-scale stochastic systems: A relaxed dissipativity approach

“…First and foremost, the proposed compositional approach here is less conservative than the one presented in [LSZ18c], in the sense that the stabilizability of individual subsystems is not necessarily required. Second, we provide a scheme for the construction of finite MDPs for a class of discrete-time nonlinear stochastic control systems whereas the construction scheme in [LSZ18c] only handles the class of linear systems. We also apply our results to a fully connected network of nonlinear systems.…”