On the Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces

Saldi, Naci; Yüksel, Serdar; Linder, Tamás

doi:10.1287/moor.2016.0832

Cited by 43 publications

(65 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Proof. The result follows from (16) and (17), the fact that W n and W are continuous at k, and Theorem 5. It can be done similar to the proof of Theorem 2, and so, we omit the details.…”

Section: Asymptotic Approximation Of Average Cost Problemsmentioning

confidence: 72%

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

Saldi

2019

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

In this paper, we consider the finite-state approximation of a discrete-time constrained Markov decision process (MDP) under the discounted and average cost criteria. Using the linear programming formulation of the constrained discounted cost problem, we prove the asymptotic convergence of the optimal value of the finite-state model to the optimal value of the original model. With further continuity condition on the transition probability, we also establish a method to compute approximately optimal policies. For the average cost, instead of using the finite-state linear programming approximation method, we use the original problem definition to establish the finitestate asymptotic approximation of the constrained problem and compute approximately optimal policies. Under Lipschitz type regularity conditions on the components of the MDP, we also obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite state space increases.

show abstract

“…Proof. The result follows from (16) and (17), the fact that W n and W are continuous at k, and Theorem 5. It can be done similar to the proof of Theorem 2, and so, we omit the details.…”

Section: Asymptotic Approximation Of Average Cost Problemsmentioning

confidence: 72%

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

Saldi

2019

IEEE Trans. Automat. Contr.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Under the assumption that the transition probability is weakly continuous in stateaction variables and setwise continuous in action variable, it was shown that if one uses sufficiently large number of grid points to partition the state space, then the resulting finite state MDP gives a near optimal policy. In [13] (i.e., full paper version of this work) a rate of convergence analysis and numerical results are also presented.…”

Section: Discussionmentioning

confidence: 99%

“…Even though in this paper we are only concerned with the asymptotic optimality of finite models, in the full paper [13], we consider also the rates of convergence; that is, we quantify how the size of the approximate finite model is related to the approximation error.…”

Section: Introductionmentioning

confidence: 99%

Finite state approximations of Markov decision processes with general state and action spaces

Saldi

Linder

Yüksel

2015

2015 American Control Conference (ACC)

Self Cite

View full text Add to dashboard Cite

The purpose of this paper is to prove existence of an ε-equilibrium point in a dynamic Nash game with Borel state space and long-run time average cost criteria for the players. The idea of the proof is first to convert the initial game with ergodic costs to an "equivalent" game endowed with discounted costs for some appropriately chosen value of the discount factor, and then to approximate the discounted Nash game obtained in the first step with a countable state space game for which existence of a Nash equilibrium can be established. From the results of Whitt we know that if for any ε > 0 the approximation scheme is selected in an appropriate way, then Nash equilibrium strategies for the approximating game are also ε-equilibrium strategies for the discounted game constructed in the first step. It is then shown that these strategies constitute an ε-equilibrium point for the initial game with ergodic costs as well. The idea of canonical triples, introduced by Dynkin and Yushkevich in the control setting, is adapted here to the game situation. 1. Introduction. We are considering a two-person Markov game over an infinite time horizon. The state space E of the process {x t } ∞ t=0 controlled by the players is taken to be a Borel space E equipped with the Borel σ-algebra E. The action spaces U 1 and U 2 of player 1 and 2, respectively, are compact subsets of some metric spaces. Let U i denote the Borel σ-algebra on U i , and let P(U i) denote the set of all probability measures on (U i , U i), 1991 Mathematics Subject Classification: Primary 90D10, 90D20; Secondary 90D05, 93E05.

show abstract

“…However, it still requires the Lipschitz continuity of some components of dynamic programs. Unlike the aforementioned approaches, the finite-state and finite-action approximation method for MDPs with σ-compact state spaces proposed by Saldi et al does not rely on Lipschitz-type continuity conditions [15,16].…”

Section: Related Workmentioning

confidence: 99%

Stochastic Subgradient Methods for Dynamic Programming in Continuous State and Action Spaces

Jang

Yang

2019

2019 IEEE 58th Conference on Decision and Control (CDC)

View full text Add to dashboard Cite

A convex optimization-based method is proposed to numerically solve dynamic programs in continuous state and action spaces. This approach using a discretization of the state space has the following salient features. First, by introducing an auxiliary optimization variable that assigns the contribution of each grid point, it does not require an interpolation in solving an associated Bellman equation and constructing a control policy. Second, the proposed method approximates the optimal value function via convex programming with a uniform convergence property in the case of convex optimal value functions. We also propose a design method for a control policy of which performance converges uniformly to the optimum as the grid resolution becomes finer in this case. Third, when a nonlinear control-affine system is considered, the convex optimization approach provides an approximate control policy with a provable suboptimality bound. Fourth, for general cases, the proposed convex formulation of dynamic programming operators can be simply modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties if a globally optimal solution to this bi-level program can be found. From our convex methods and analyses, we observe that convexity in dynamic programming deserves attention as it can play a critical role in obtaining a tractable and convergent numerical solution.

show abstract

On the Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces

Cited by 43 publications

References 45 publications

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

Finite state approximations of Markov decision processes with general state and action spaces

Stochastic Subgradient Methods for Dynamic Programming in Continuous State and Action Spaces

Contact Info

Product

Resources

About