Discrete versions of an algorithm due to Varaiya

A First Course in Stochastic Models

“…These authors extended the original value-iteration bounds of MacQueen (1966) for the discounted cost case to the average cost case. The modified value-iteration algorithm with a dynamic relaxation factor comes from Popyack et al (1979). The first proof of the geometric convergence of the undiscounted value-iteration algorithm was given by White (1963) under a very strong recurrence condition.…”

A New Value Iteration method for the Average Cost Dynamic Programming Problem

“…we see that if 7yk = 1 for all k, the new value iteration (7)-(8) becomes similar to the known value iteration (9)-(10): the updating formulas are the same in both methods, but the order of updating A is just reversed relatively to the order of updating h. We note that there is also a variant of the standard method (9)-(10) that involves interpolations between hk and hk+1 according to a stepsize parameter (see [Sch71], [Pla77], [Var78], [PBW79], [Put94], [Ber95]). However, the new method does not seem as closely related to this variant.…”

A new policy iteration scheme for Markov decision processes using Schweitzer's formula

“…This can be done by linear programming (LP) [3], [8], [12], [17], value iteration [4], [7], [9], [16], [18], [23], [25] or policy iteration [9], [10], [11]. Policy iteration was first proposed by Howard [9].…”