On the total reward variance for continuous-time Markov reward chains

Abstract

As an extension of the discrete-time case, this note investigates the variance of the total cumulative reward for continuous-time Markov reward chains with finite state spaces. The results correspond to discrete-time results. In particular, the variance growth rate is shown to be asymptotically linear in time. Expressions are provided to compute this growth rate.

Risk-sensitive average optimality in Markov decision processes

“…Necessary and sufficient conditions guaranteeing solutions of (26), (27), (28) are discussed in the next section. Now we are in a position to formulate necessary and sufficient average reward optimality conditions for the risk sensitive models if optimality equations (26), (27), (28) are fulfilled. Recall that the discrepancy functionφ x k ,x k+1 (w, g) := r ij − g + w j − w i .…”

“…In what follows we present necessary and sufficient conditions for the existence of a solution of optimality equations (26)- (28) for unichain models as well as for a very specific case of multi-chain models.…”

“…and (27), (28) can be rewritten as the following sets of nonlinear equations (herev i := e γŵi , v * i := e γw * i ,ρ = e γĝ , ρ * := e γg * )…”

“…(26) follows immediately from the Perron-Frobenius theorem. Using policy iterations we can also verify (27) and (28), see [25] for details.…”

An Inequality for Variances of the Discounted Rewards

“…In particular there is a significant amount of literature on various optimization of Markov Decision Processes (MDPs), see the pioneering work by Jaquette [4] and Sobel [7,8,9], a survey by White [11], and recent references by Van Dijk and Sladký [10] and Baykal-Gürsoy and Gürsoy [1].…”