Discounted semi‐Markov decision processes: linear programming and policy iteration

Abstract

For semi‐Markov decision processes with discounted rewards we derive the well known results regarding the structure of optimal strategies (nonrandomized, stationary Markov strategies) and the standard algorithms (linear programming, policy iteration). Our analysis is completely based on a primal linear programming formulation of the problem.

Solving stochastic dynamic programming problems by linear programming — An annotated bibliography

“…These models can be treated by linear programming, too, as was shown by Fox [1966], Denardo/ Fox [1968], Osaki/Mine [1968], Denardo [1970], Hinomoto [1971] (who started from a management control problem) and Wessels/van Nunen [1975] (whose analysis is completely based on a primal linear programming formulation). Mine/Tabata [1970] at first transformed a continuous parameter model into a discrete one and then applied a linear programming method.…”

A set of successive approximation methods for discounted Markovian decision problems

“…As known, [BlackweU, 1962], [Wessels and van Nunen, 1975], we can restrict ourselves to nonrandomized stationary policies, which will be denoted by f6K := {K(1) x K(2) x ... x K(N)}. The components us(i) of the N x 1 vector u s give the total expected discounted reward if the initial state is i and policy f is used.…”

Stopping Times and Markov Programming

“…Hence for this new Markov decision process attention may be restricted to memoryless strategies (e.g. [6J,[IOJ,[12J), which implies the same for the original problem. This new Markov decision process is defined in the following way: S, the new set of states, consists of So and two representations of S: S* = {s* I s E S} and S* = {s* I s € S}.…”