Exponential bounds for discrete-time singularly perturbed Markov chains

Abstract

This paper develops exponential type upper bounds for scaled occupation measures of singularly perturbed Markov chains in discrete time. By considering two-time scale in the Markov chains, asymptotic analysis is carried out. The cases of the fast changing transition probability matrix is irreducible and that are divisible into l ergodic classes are examined first; the upper bounds of a sequence of scaled occupation measures are derived. Then extensions to Markov chains involving transient states and/or nonhomogeneous transition probabilities are dealt with. The results enable us to further our understanding of the underlying Markov chains and related dynamic systems, which is essential for solving many control and optimization problems.  2004 Elsevier Inc. All rights reserved.

Implied Markov transition matrices under structural price models

“…In the optimal control literature, decisions variables that appear in the dual linear programming formulation of Markov Decision Processes represent state occupation measures, see (Derman and Klein 1965, Mine and Tabata 1970, de Farias and Van Roy 2003. Similarly, state occupation measures appear in linear programming formulations of constrained Markov decision processes (Feinberg andRothblum 2012, Lee et al 2014), in convex analytic methods for Markov decision processes (Bhatt and Borkar 1996, Borkar 2002, Haskell and Jain 2015, in the perturbation theory of Markov chains (Zhang and Yin 2004). In Stockbridge (2004), the pricing of barrier options and lookback options is formulated as a linear programming problem where dual variables represent occupation measures.…”