Action Time Sharing Policies for Ergodic Control of Markov Chains

Abstract

Ergodic control for discrete time controlled Markov chains with a locally compact state space and a compact action space is considered under suitable stability, irreducibility and Feller continuity conditions. A flexible family of controls, called action time sharing (ATS) policies, associated with a given continuous stationary Markov control, is introduced. It is shown that the long term average cost for such a control policy, for a broad range of one stage cost functions, is the same as that for the associated stationary Markov policy. In addition, ATS policies are well suited for a range of estimation, information collection and adaptive control goals. To illustrate the possibilities we present two examples: The first demonstrates a construction of an ATS policy that leads to consistent estimators for unknown model parameters while producing the desired long term average cost value. The second example considers a setting where the target stationary Markov control q is not known but there are sampling schemes available that allow for consistent estimation of q. We construct an ATS policy which uses dynamic estimators for q for control decisions and show that the associated cost coincides with that for the unknown Markov control q.

Learning Deep Mean Field Games for Modeling Large Population Behavior

“…Methods in unknown MDP estimation and inverse reinforcement learning aim to learn an optimal policy while estimating an unknown quantity of the MDP, such as the transition law (Burnetas & Katehakis, 1997), secondary parameters (Budhiraja et al, 2012), and the reward function (Ng & Russell, 2000). The maximum entropy IRL framework has proved successful at learning reward functions from expert demonstrations (Ziebart et al, 2008;Boularias et al, 2011;Kalakrishnan et al, 2013).…”