Compositional reasoning for weighted Markov decision processes

Abstract

h i g h l i g h t s• We develop compositional methods for reasoning about weighted Markov decision processes (MDPs).• A coinductive simulation-based behavioural preorder is proposed for weighted MDPs.• The preorder is preserved by structural operators for constructing weighted MDPs from components.• For finitary convergent processes, the preorder can be characterised by a probabilistic logic and a novel form of testing. a r t i c l e i n f o b s t r a c tWeighted Markov decision processes (MDPs) have long been used to model quantitative aspects of systems in the presence of uncertainty. However, much of the literature on such MDPs takes a monolithic approach, by modelling a system as a particular MDP; properties of the system are then inferred by analysis of that particular MDP. In contrast in this paper we develop compositional methods for reasoning about weighted MDPs, as a possible basis for compositional reasoning about their quantitative behaviour. In particular we approach these systems from a process algebraic point of view. For these we define a coinductive simulation-based behavioural preorder which is compositional in the sense that it is preserved by structural operators for constructing weighted MDPs from components.For finitary convergent processes, which are finite-state and finitely branching systems without divergence, we provide two characterisations of the behavioural preorder. The first uses a novel quantitative probabilistic logic, while the second is in terms of a novel form of testing, in which benefits are accrued during the execution of tests.

Verification of Markov Decision Processes Using Learning Algorithms

“…Compositional techniques aim to first analyse parts of the system separately and combine the sub-results to obtain an overall result, e.g. [40,59,78,20,42,21]. Then, there are abstraction approaches which try to merge states with equivalent or sufficiently similar behaviour w.r.t.…”

Trust domains in system models: algebra, logic, utility, and combinators

“…Markov Chains are studied in [26]: they support reasoning about complex notions such as average utility with a given time discount, but do not provide compositionality results over model structures. Process calculi for Markov decision processes, which include both stochastic and cost-based decision-making, provide such compositionality results for the class of systems that do not permit negative utility, and then only for a notion of simulation [17]. It may be possible to extend work on probabilistic synchronous calculi [3] to provide a process calculus with more general notions of cost, and with full bisimulation.…”

Modelling and Analysis of Markov Reward Automata

“…The underlying philosophy is to already reduce on the specification, not having to first generate a large model before being able to minimise. The parallel composition of MRAs is described in the technical report [19] and is equivalent to [11] for the probabilistic transitions.…”