Use of gain and other estimates to obtain sub-optimal solution of a class of Markov decision processes†

Abstract

Studies in the past decade have indicated considerable interest in the problems of optimizing processes having Merkoviun property. Certain properties of the transition matrix, associated with such processes, regarding the estimates of the steady-state probability distribution, gain estimate, their order of approximation, error in estimato etc., has been considered in some depth in this paper. The computational ease of these estimates have led to the development of a direct search type Buboptimal algorithm. The algorithm has been employed to Howard's Taxi-Cab problem and Hamza'a problem of coupled Mar-kov processes. Often, this sub-optimal solution is itself the optimal solution, which interestingly is the case for tho problems solved in this paper. In case where a strictly optimal solution is necessary, the Howard's algorithm may be persued with the SUb-optimal decision as the starting policy. This will still be advantageous from the point of view of computation time for small-scale systems. IntroductionStudies in the past decade have indicated considerable and continuing interest in the problems of optimizing processes having the Markovian property. An important structure of dynamic programming was first formulated by Bellman (1957) and then extensively developed by Howard (1960) for the optimization of Markov decision processes. Some more analysis and numerous applications in this area have been reported since then.In these processes, the system makes Markovian transition from one state to another belonging to a finite set of states. Associated with each transition is a reward obtained from a reward structure which is accumulated after each transition. The probabilities P = [Pij] and the rewards R = [rij] incurred depend upon the decision d = [k i ] made from a finite number of alternatives.Certain properties of the probability matrix P attracted the attention of the present author and Mustafi (1970) regarding estimation of steady-state probability distribution. In the present paper a method of estimating the steadystate probability distribution, the gain and their order of approximation will be considered in some detail. The computational ease of these estimates allows a different and rather direct approach to some of the optimization problems of Howard (1960) and that of Hamza (1965). This approach, though sub-optimal in nature, has some advantage for small-scale process, at least in respect of computer time. Often this sub-optimal solution is itself the optimal solution. \Vhen, however, the system is a large-scale one, the method runs into difficulties. Unfortunately, Howard's method also becomes time-consuming. In such cases, a modified Howard's algorithm using Monte Carlo games could be advantageous in determining a sub-optimal policy. Here also, the gain estimates play an important role.A rather direct method for determining the sub-optimal decision, as mentioned earlier, will also be presented in this paper. However, only regular