Policy Synthesis for Collective Dynamics

Abstract

In this paper we consider the problem of policy synthesis for systems of large numbers of simple interacting agents where dynamics of the system change through information spread via broadcast communication. By modifying the existing modelling language Carma and giving it a semantics in terms of continuous time Markov decision processes we introduce a natural way of formulating policy synthesis problems for such systems. However, solving policy synthesis problems is difficult since all non-trivial models result in very large state spaces. To combat this we propose an approach exploiting the results on fluid approximations of continuous time Markov chains to obtain estimates of optimal policies.

Fluid Approximation–based Analysis for Mode-switching Population Dynamics

“…In this section we define a class of stochastic population models for modelling systems where the dynamics of a population can be separated into discrete modes of behaviour. In [30] we argued that such models arise naturally from broadcast communication in collective systems with modes corresponding to information that the collective has about its operating environment. To start off we extend our running example to give the following motivating example.…”

“…In previous our work [30], such changes in the macro-level behaviour were identified as mode-switching population dynamics, a particular form of hybrid model. In this case, levels of knowledge and the related dynamics correspond to the different dynamic modes of the population system.…”

A Case Study of Policy Synthesis for Swarm Robotics

“…The Function Labelled Transition Systems (FuTS) style operational semantics [10] of Carma, as described in [18], give rise to a labelled transition system which can be simulated directly or translated into a CTMC for numerical analysis. The aim of this section is to introduce the extension Carma-C [22] for specifying policy synthesis problems. The underlying CTMDP model allows us to specify non-deterministic behaviour, corresponding to different possible action choices, in a Carma-C model.…”

“…Further details on the decision process semantics are omitted here due to space constraints and can be found in [22]. We simply note that the state of the pCTMDP corresponding to the described Carma-C model are represented by one counting variable for each considered location.…”

“…Swarm robots provide rich modelling examples in the context of collective systems, since the directly controllable behaviours and interactions are those of individual robots but the design goals for the systems are phrased in terms of the aggregate behaviour of the entire collective rather that of individual robots. To that end, Carma can be equipped with continuous time Markov decision processes (CT-MDPs) based semantics [22] providing a natural formalisation of policy synthesis problems and bridging the gap between the formal high-level modelling and policy synthesis problems for collectives such as robot swarms. The contribution of this paper is to illustrate these constructions with a case study and show how ideas from formal modelling and policy synthesis come together in a framework for stochastic control or parameter synthesis problems.…”