The Optimal Reward Operator in Negative Dynamic Programming

Maitra, A.; Sudderth, William D.

doi:10.1287/moor.17.4.921

Cited by 12 publications

(30 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For a proof of the preceding theorem, see Zinsmeister [42], which also showed that the uniform analyticity condition in (ii) implies the monotonicity of Ψ for analytic sets. According to [24,42], this theorem is a special case of a very general result of Moschovakis; see [24,42] for related references.…”

Section: C1 Ordinals and Transfinite Value Iterationmentioning

confidence: 97%

“…7.21]. Equivalently, f is lower semi-analytic if and only if its epigraph {(y, c) | f (y) ≤ c, y ∈ D, c ∈ (−∞, +∞)} is analytic (see [2, p. 186], [24]). 4 For two Borel spaces Y, Z, a universally measurable stochastic kernel on Y given Z is by definition a universally measurable mapping from Z to the space P(Y ).…”

Section: Borel-spaces Mdp and The Total Cost Model (Gc)mentioning

confidence: 99%

Section: C1 Ordinals and Transfinite Value Iterationmentioning

confidence: 99%

“…Lemma C.1 can be established by using the proof arguments given in Maitra and Sudderth's analysis of transfinite value iteration for the positive costs model [24]. In the rest of this appendix, we will explain the main proof steps-we do not include a detailed proof because most arguments are the same as in [24]. (The paper [24] considers the reward maximization framework and the negative rewards model, which is equivalent to the positive costs model in the cost minimization framework we use.…”

Section: C1 Ordinals and Transfinite Value Iterationmentioning

confidence: 99%

“…However, to apply the theorem, we also need to define Ψ for all subsets in S × [0, +∞], including the epigraphs of those non-measurable functions. This is achieved in [24] by using an outer-integral formulation to define Ψ. Specifically, for any set E ⊂ S × [0, +∞], we define…”

Section: (C2)mentioning

confidence: 99%

See 4 more Smart Citations

On Convergence of Value Iteration for a Class of Total Cost Markov Decision Processes

Yu¹

2015

SIAM J. Control Optim.

View full text Add to dashboard Cite

We consider a general class of total cost Markov decision processes (MDP) in which the one-stage costs can have arbitrary signs, but the sum of the negative parts of the one-stage costs is finite for all policies and all initial states. This class, which we refer to as the general convergence (GC) total cost model, contains several important subclasses of problems, e.g., positive costs problems, bounded negative costs problems, and discounted problems with unbounded onestage costs. We study the convergence of value iteration for the (GC) model, in the Borel MDP framework with universally measurable policies. Our main results include (i) convergence of value iteration when starting from certain functions above the optimal cost function; (ii) convergence of transfinite value iteration starting from zero, as well as convergence of ordinary nontransfinite value iteration for finite control or certain finite state (GC) problems, in the special case where the optimal cost function is nonnegative; and (iii) partial convergence of value iteration starting from zero, for a subset of initial states. These results extend several previously known results about the convergence of value iteration for either positive costs problems or (GC) total cost problems. Introduction.In this paper we study convergence properties of value iteration for a class of Markov decision processes (MDP) under the undiscounted total cost criterion. Specifically, we consider problems in which the one-stage costs can take both positive and negative values, but we assume that under any policy and for any starting state, the expected total sum, over the infinite horizon, of the negative parts of the one-stage cost is finite. Following the extensive survey on this class of MDP (Feinberg [15]), we shall refer to this total cost model as the general convergence total cost model (GC). It contains several classes of special models, in particular, the positive costs model (P), where all the one-stage costs are nonnegative, the negative costs model (N) where all the one-stage costs are nonpositive and the optimal costs are finite, as well as the bounded negative costs model [28, Chap. 7.2] and a subset of the discounted models with unbounded costs (UD).It is known that for the (GC) model, value iteration starting from the constant function zero need not converge to the optimal cost function, surprisingly, even for finite state and control problems (see van der Wal [37, Ex. 3.2], Feinberg [15, Ex. 6.10]). This is also true for the positive costs model (P) (although convergence is ensured for (P) if each state has only a finite number of feasible controls [2, Prop. 9.18] or more generally, if certain compactness conditions [1], [2, Prop. 9.17] or semicontinuity and compactness model assumptions [29] are satisfied). In the (P) case, Maitra and Sudderth established the convergence of transfinite value iteration [24], and Whittle formulated a sufficient condition, called the bridging condition, for the

show abstract