Sensitivity analysis approaches to high-dimensional screening problems at low sample size

Abstract

Sensitivity analysis is an essential tool in the development of robust models for engineering, physical sciences, economics and policymaking, but typically requires running the model a large number of times in order to estimate sensitivity measures. While statistical emulators allow sensitivity analysis even on complex models, they only perform well with a moderately low number of model inputs: in higher dimensional problems they tend to require a restrictively high number of model runs unless the model is relatively linear. Therefore, an open question is how to tackle sensitivity problems in higher dimensionalities, at very low sample sizes. This article examines the relative performance of four sampling-based measures which can be used in such high-dimensional nonlinear problems. The measures tested are the Sobol' total sensitivity indices, the absolute mean of elementary effects, a derivative-based global sensitivity measure, and a modified derivative-based measure. Performance is assessed in a 'screening' context, by assessing the ability of each measure to identify influential and non-influential inputs on a wide variety of test functions at different dimensionalities. The results show that the best-performing measure in the screening context is dependent on the model or function, but derivative-based measures have a significant potential at low sample sizes that is currently not widely recognised.

Challenges and Lessons Learned in Applying Sensitivity Analysis to Building Stock Energy Models

“…Sample sizes of 20, 100 &1000 buildings for each method. SimStock (C1) Sobol' method, (C2) Derivative-based Global Sensitivity Measure (DGSM) (Becker et al, 2018), (C3) Elementary Effects (Morris Method) (Campolongo et al, 2007), 51 variable parameters in an urban district in North London. Each method evaluated at 10, 20 and 50 estimates.…”

A comparison of performance of three variance-based sensitivity analysis methods on an urban-scale building energy model

“…Values close to one indicate strong agreement between the two rankings. • Screening performance - Becker et al (2018) propose the number of parameters wrongly identified as influential as a fraction of the number of influential parameters as a test of the accuracy of screening. In this study, wrongly excluding influential parameters is considered less desireable than wrongly including non-influential parameters and so the fraction of false negatives is also calculated.…”

“…• Derivative based (DGSM) (Becker et al, 2018) -Similar to EER, this method uses a smaller increment.…”

“…The DGSM method performs least well of all those considered. This seems to be due to the underlying nature of the model - Becker et al (2018) demonstrated that allow the DGSM method performs well on smooth functions, the small increment results in poor performanc in step-functions. It is likely that the setpoint temperature regime for heating and cooling demand represents a significant enough step to make this method unsuitable for use with building energy simulation.…”

VISCOUS: A Variance‐Based Sensitivity Analysis Using Copulas for Efficient Identification of Dominant Hydrological Processes

“…Another eminent challenge is that these emulation techniques usually work well when the problem at hand has a relatively low number of factors. But when the dimensionality of the problem is reasonably high, as in more complex models, emulators can become practically unsuitable for finding influential factors due to the curse of dimensionality and over‐fitting issues (Becker et al., 2018). Emulation‐based GSA suffers from two more drawbacks: (a) choosing the best emulator may not be easy due to the existence of a plethora of different emulators, and (b) some emulators also need ad‐hoc experimental designs and specific arrangement of sample points for the sake of efficiency.…”