Denumerable state markovian sequential control processes: On randomizations of optimal policies

Abstract

We consider a denumerable state Markovian sequential control process. It is well known that when we consider the expected total discounted income as a criterion, there exists a nonrandomized stationary policy that is optimal. It is also well known that when we consider the expected average income as a criterion, an optimal nonrandomized stationary policy exists when a certain system of equations has a solution. The problem considered here is: if there exist two optimal nonrandomized stationary policies, will a randomization of these two policies be optimal? It is shown that in the discounted case the answer is always yes, but in the average income case, the answer is yes only under certain additional conditions.

Constrained Discounted Markov Decision Chains

“…PROOF: The first statement is proved in Chitgopekar [6] for countable state spaces and bounded rewards, and the randomization is allowed to be both state and time dependent. Let d(p) = (p,f,e) be as above.…”

“…The final step of the proof in Chitgopekar [6] depends on the boundedness of the rewards. We give a different proof.…”

Discounted Cost Markov Decision Processes with a Constraint

“…I thank Professor Masami Kurano of Chiba University for informing me of a reference [5] and for helpful advice. I also thank the referee for comments that have helped to improve this paper.…”