Reducing Simulation Input-Model Risk via Input Model Averaging

Abstract

Input uncertainty is an aspect of simulation model risk that arises when the driving input distributions are derived or “fit” to real-world, historical data. Although there has been significant progress on quantifying and hedging against input uncertainty, there has been no direct attempt to reduce it via better input modeling. The meaning of “better” depends on the context and the objective: Our context is when (a) there are one or more families of parametric distributions that are plausible choices; (b) the real-world historical data are not expected to perfectly conform to any of them; and (c) our primary goal is to obtain higher-fidelity simulation output rather than to discover the “true” distribution. In this paper, we show that frequentist model averaging can be an effective way to create input models that better represent the true, unknown input distribution, thereby reducing model risk. Input model averaging builds from standard input modeling practice, is not computationally burdensome, requires no change in how the simulation is executed nor any follow-up experiments, and is available on the Comprehensive R Archive Network (CRAN). We provide theoretical and empirical support for our approach.

A bi‐level ensemble learning approach to complex time series forecasting: Taking exchange rates as an example

“…One limitation on prediction accuracy relates to how to integrate the results of individual models in the ensemble forecasting methodology. In other words, it is important to construct ensemble strategy, which has significant impacts on the prediction performance (Nelson et al, 2021). As shown in Table 2, the ensemble strategies can be divided into linear or nonlinear and equal weight or non-equal weight categories.…”

Global Sensitivity Analysis with Mixtures: A Generalized Functional ANOVA Approach

“…We illustrate first a result 2 The weights do not necessarily have the interpretation of a second-order distribution over (F, B(F )). In Nelson et al (2021), the probabilities represent the optimal weight of a convex combination of the F q X distributions, so that the resulting mixture is optimal with respect to a quadratic score when fitted to available data. proven in Borgonovo et al (2018).…”

“…Several possibilities have been studied to aggregate a set of plausible distributions into a mixture. A mixture distribution can be assigned through the use of Bayesian model averaging Nannapaneni and Mahadevan (see 2016), or, as in Nelson, Wan, Zou, Zhang, and Jiang (2021), by finding the weights that ensure a best fit to the data, or through a linear aggregation rule if the analyst elicits prior information on the simulator inputs from expert opinions. (The aggregation of expert opinions is a vast subject and we refer to O' Hagan et al (2006), Oakley and O'Hagan (2007), Cooke (2013), and Oppenheimer, Little, and Cooke (2016), on alternative methodologies.…”

“…In Nelson et al. ( 2021 ), the probabilities represent the optimal weight of a convex combination of the distributions, so that the resulting mixture is optimal with respect to a quadratic score when fitted to available data.…”

Comparing Data Collection Strategies via Input Uncertainty When Simulating Testing Policies Using Viral Load Profiles

“…For recent advancements see Lam and Qian (2017), Lam and Qian (2018). Efforts by Nelson et al (2020) have also been made to reduce input uncertainty using frequentist modelling averaging. Morgan et al (2019) recently considered detection and quantification of bias caused by input modelling.…”