Reverse sensitivity testing: What does it take to break the model?

Abstract

Sensitivity analysis is an important component of model building, interpretation and validation. A model comprises a vector of random input factors, an aggregation function mapping input factors to a random output, and a (baseline) probability measure. A risk measure, such as Value-at-Risk and Expected Shortfall, maps the distribution of the output to the real line. As is common in risk management, the value of the risk measure applied to the output is a decision variable. Therefore, it is of interest to associate a critical increase in the risk measure to specific input factors. We propose a global and model-independent framework, termed 'reverse sensitivity testing', comprising three steps: (a) an output stress is specified, corresponding to an increase in the risk measure(s); (b) a (stressed) probability measure is derived, minimising the Kullback-Leibler divergence with respect to the baseline probability, under constraints generated by the output stress; (c) changes in the distributions of input factors are evaluated. We argue that a substantial change in the distribution of an input factor corresponds to high sensitivity to that input and introduce a novel sensitivity measure to formalise this insight. Implementation of reverse sensitivity testing in a Monte-Carlo setting can be performed on a single set of input/output scenarios, simulated under the baseline model. Thus the approach circumvents the need for additional computationally expensive evaluations of the aggregation function. We illustrate the proposed approach through a numerical example of a simple insurance portfolio and a model of a London Insurance Market portfolio used in industry.

Scenario Weights for Importance Measurement (SWIM) – An R Package for Sensitivity Analysis

“…Complex quantitative models are used extensively in actuarial and financial risk management applications, as well as in wider fields such as environmental risk modelling Borgonovo and Plischke, 2016;Pesenti et al, 2019). The complexity of such models (high dimensionality of inputs; non-linear relationships) motivates the performance of sensitivity analyses, with the aim of providing insight into the ways that model inputs interact and impact upon the model output.…”

“…The SWIM approach is largely based on Pesenti et al (2019) and uses theoretical results on risk measures and sensitivity measures developed in that paper. An early sensitivity analysis approach based on scenario weighting was proposed by Beckman and McKay (1987).…”

“…( 1) In what follows, we denote by a superscript W quantities of interest under the stressed model, such as F W , E W for the probability distribution and expectation under the stressed model, respectively. We refer to Pesenti et al (2019) and references therein for further mathematical details and derivations of solutions to (1). Table 2 provides a collection of all implemented types of stresses in the SWIM package.…”

“…The precise constraints of (1) are explained below. (7) The solutions to the optimisations (2) and (6) are worked out fully analytically (Pesenti et al, 2019), whereas problems (3), (4), (5) and (7) require some root-finding. Specifically, problems (3), (4) and (5) rely on the package nleqslv, whereas (7) uses the uniroot function.…”

“…The SWIM package implements a flexible sensitivity analysis framework, based primarily on results and tools developed by Pesenti et al (2019). SWIM provides a stressed version of a stochastic model, subject to model components (random variables) fulfilling given probabilistic constraints (stresses).…”