Adaptive Importance Sampling Technique for Markov Chains Using Stochastic Approximation

Abstract

For a discrete-time finite-state Markov chain, we develop an adaptive importance sampling scheme to estimate the expected total cost before hitting a set of terminal states. This scheme updates the change of measure at every transition using constant or decreasing step-size stochastic approximation. The updates are shown to concentrate asymptotically in a neighborhood of the desired zero variance estimator. Through simulation experiments on simple Markovian queues, we observe that the proposed technique performs very well in estimating performance measures related to rare events associated with queue lengths exceeding prescribed thresholds. We include performance comparisons of the proposed algorithm with existing adaptive importance sampling algorithms on a small example. We also discuss the extension of the technique to estimate the infinite horizon expected discounted cost and the expected average cost. * Work supported in part by grant III.5(157)/99

Approximating zero-variance importance sampling in a reliability setting

“…Thus, the dominant paths have probability Θ (1) and the non-dominant paths have probability o (1). This implies that p c (y) → 1, and then that p 0 → 1, when ε → 0.…”

“…. , c. We assume that each component is either in a failed state or in an operational state, and that the system evolves as a CTMC whose state is a vector y = (y (1) , . .…”

“…We will assume henceforth that v matches µ at 0 ′ and in F , where its value is known: v(0 ′ ) = 0 and v(y) = 1 for y ∈ F . Several types of adaptive IS methods that learn a good function v iteratively have been proposed and studied in the literature; see [1,4,14,15,20,24,28], for example. The general idea of these adaptive methods is to use the realizations of the previous sample paths, where each sample path provides one realization of X, to construct an approximation v. Under some conditions, this approximation can sometimes be shown to converge to µ as n → ∞, where n is the number of realizations of X.…”

“…As an extreme case, taking E = U means that we have a direct estimate of µ(y) for all states y ∈ U , and no interpolation is needed. Then, we are back to an estimator similar to that of [1], depending on how the estimation is done (these authors change the measure dynamically at each step, at the same time as they update their estimates of µ(y)). Our proposal is a matter of compromise between this extreme case and just taking v 0 (the other extreme).…”

“…Let the transitions probabilities of the corresponding DTMC be those given in Figure 1, where the states in F are shaded, and state y = (y (1) , y (2) ) means (as usual) that y (1) components of type 1 and y (2) components of type 2 are down.…”