Experimental Design for Sensitivity Analysis, Optimization, and Validation of Simulation Models

Abstract

This chapter gives a survey on the use of statistical designs for what-if analysis in simulation, including sensitivity analysis, optimization, and validation/verification. Sensitivity analysis is divided into two phases. The first phase is a pilot stage, which consists of screening or searching for the important factors among (say) hundreds of potentially important factors. A novel screening technique is presented, namely sequential bifurcation. The second phase uses regression analysis to approximate the input/output transformation that is implied by the simulation model; the resulting regression model is also known as a metamodel or a response surface. Regression analysis gives better results when the simulation experiment is well designed, using either classical statistical designs (such as fractional factorials) or optimal designs (such as pioneered by Fedorov, Kiefer, and Wolfowitz). To optimize the simulated system, the analysts may apply Response Surface Methodology (RSM); RSM combines regression analysis, statistical designs, and steepest-ascent hill-climbing. To validate a simulation model, again regression analysis and statistical designs may be applied. Several numerical examples and case-studies illustrate how statistical techniques can reduce the ad hoc character of simulation; that is, these statistical techniques can make simulation studies give more general results, in less time. Appendix 1 summarizes confidence intervals for expected values, proportions, and quantiles, in terminating and steady-state simulations. Appendix 2 gives details on four variance reduction techniques, namely common pseudorandom numbers, antithetic numbers, control variates or regression sampling, and importance sampling. Appendix 3 describes jackknifing, which may give robust confidence intervals.

Investigating the Effectiveness of Variance Reduction Techniques in Manufacturing, Call Center and Cross-Docking Discrete Event Simulation Models

“…The following literature with extensive bibliographies is recommended to readers interested in going further into the subject i.e. (Nelson 1987), (Kleijnen 1988) and . In next section is a discussion on the three variance reduction techniques that appear to have the most promise of successful application to discrete event simulation modeling is presented.…”

State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments

“…Sacks et al (1989) classify problems for simulation analysts as prediction, calibration, and optimization. Kleijnen (1998) distinguishes among global (not local) sensitivity analysis, optimization, and validation of simulation models. (In global sensitivity analysis the simulation inputs vary over the whole experimental area, rather than infinitesimally.)…”

An Overview of the Design and Analysis of Simulation Experiments for Sensitivity Analysis

“…Continue to apply the OLS point estimator (10), but use the covariance formula (11) instead of (13) ii. Switch from OLS to Generalized Least Squares (GLS) with estimated ) (w cov based on m > n replications (using different PRN); for details see Kleijnen (1992Kleijnen ( , 1998). …”

“…An update is Kleijnen (1998). A bird-eye's view of DOE in simulation is Kleijnen et al (2003a), which covers a wider area than this review-without using any equations, tables, or figures; this review covers a smaller area-in more detail.…”