First-order sensitivity of the optimal value in a Markov decision model with respect to deviations in the transition probability function

Abstract

Markov decision models (MDM) used in practical applications are most often less complex than the underlying 'true' MDM. The reduction of model complexity is performed for several reasons. However, it is obviously of interest to know what kind of model reduction is reasonable (in regard to the optimal value) and what kind is not. In this article we propose a way how to address this question. We introduce a sort of derivative of the optimal value as a function of the transition probabilities, which can be used to measure the (first-order) sensitivity of the optimal value w.r.t. changes in the transition probabilities. 'Differentiability' is obtained for a fairly broad class of MDMs, and the 'derivative' is specified explicitly. Our theoretical findings are illustrated by means of optimization problems in inventory control and mathematical finance.

A concept of copula robustness and its applications in quantitative risk management

“…The largest generator of d M is called the maximal generator of d M and will be denoted by [35,Definition 3.1]. Examples for d M and M are discussed in Kern et al [26] and Müller [34,35]. Now denote by P ψ the set of all transition functions P = (P n ) N −1 n=0 ∈ P with P n ((x, a), • ) ∈ M ψ 1 (E) for all (x, a) ∈ D n and n = 0, .…”

“…where we set inf ∅ := ∞. Examples for M and M are discussed in Kern et al [26] and Müller [34]. In the following result, we assume that ψ is a bounding function for any Q ∈ P ψ .…”

“…x n (P ) := V P n (x). The set M Höl,α separates points in M ψ α 1 (R + ), implying that d M Höl,α (defined by (6.2) with M := M Höl,α ) provides a metric on M ψ α 1 (R + ); see Kern et al [26] for details. Let d M Höl,α ,ψ α be defined by (6.3) with M := M Höl,α and ψ := ψ α .…”