Algorithms for singularly perturbed limiting average Markov control problems

Abstract

I n t r o d u c t i o nIn this paper we consider a singularly perturbed Markov Decision Process with the limiting average cost criterion. We assume that the underlying process is composed of n separate irreducible processes, and that the small perturbation is such that it "unites" these processes into a single irreducible process. This structure corresponds to the Markov chains admitting "strong and weak interactions" that arises in many applications, and was studied by a number of authors (e.g., see Delebecque and Quadrat [5] In Section 2 we introduce the formulation and some results given by Bielecki and Fi1a.r [2] of the underlying control problem for the singula.rly perturbed MDP; the so-called "limit Markov Control Problem'' (limit MCP). In particular these authors proved that an optimal solution to the perturbed MDP can be approximated by an optimal solution of the limit MCP for sufficiently small perturbation.In Sectioii 3 we demonstrate tha.t the above limit Markov Control Problem ca.n be solved by a suitably constructed linear program.In Section 4 we construct an algorithm for solving the limit Markov Control Problem based on the policy improvement method. Recently we learned that this algorithm is similar to one given by Pervoewnskii and Gaitsgori [12]. However, these authors did not explicitly consider the limit Markov Control Problem, and worked only in t,he smaller class of deterministic strategies. With every A E IT we shall associate the following quantities: Definitions and Preliminaries. ( The "classical" limiting average Markov Decision problem is the optimization problem: Find x 0 E lI such that J(s,aO) = m;xJ(s,n) for dl s E S.A stra.tegy K O satisfying the above will be called optimal. It is well known that there always exists an optimal deterministic policy and there is a number of finite algorithms for its computation (e.g., Dena.rdo [6], Derman (71, Kallenberg [8]).In this paper we shall assume that:( A l ) S = u:,,S; where Si n Sj = 8 if i # j, n > I, cards; = n;, nl + . . . Consequently we can think of r as being the "union" of n smaller MDP's r;, defined on the state space Si, for each i = 1 , 2 , . , . ,n, respectively. Note that if IIi is the space of stationary strategies in r;, then a strategy A E IT in I? can be written in the natural way as K = (K', r2,. . . , +' ), where xi E IIi. The probability transition mat,rix in ri corresponding to ri is, of course, defined by: P;(ri) := ( p # +~( x~) ) # ,~,~~, , and the generator G;(n') and the Cesaro-limit P;(ri) matrices can be defined in a manner analogous to that in the original process I?. In addition, we assume that:'Not? l.hat action z E A ( s ) may not be the same as action i E A(s') if "'The nointion [ u / ,~ will be tlsed to denote the s-th entry of a vector U s # SI. This simplification of notal.ion shoold not cause ambiguity.

Robust Control of Markov Decision Processes with Uncertain Transition Matrices

“…Our work is motivated by the fact that in many practical problems, the transition matrices have to be estimated from data, and this may be a difficult task; see, for example, Kalyanasundaram et al (2001), Feinberg and Shwartz (2002), Abbad and Filar (1992), and Abbad et al (1992). It turns out that estimation errors may have a huge impact on the solution, which is often quite sensitive to changes in the transition probabilities.…”

Exponential bounds for discrete-time singularly perturbed Markov chains

“…Some of the recent development on ✩ The research was supported in part by the National Science Foundation. the subject and its various applications, we refer the reader to [1][2][3][4][5]12,14,18]. For related numerical results and simulations of singularly perturbed Markov chains and the associated control problems, see [9,19].…”

Two-Time Scale Controlled Markov Chains: A Decomposition and Parallel Processing Approach

“…It has been shown in [10] that, when ε tends to zero, the solution of the MDP can be approximated by the solution of a limit control problem (LCP). To this LCP corresponds the…”