ASAP3: a batch means procedure for steady-state simulation analysis

Abstract

We introduce ASAP3, a refinement of the batch means algorithms ASAP and ASAP2, that delivers point and confidence-interval estimators for the expected response of a steady-state simulation. ASAP3 is a sequential procedure designed to produce a confidence-interval estimator that satisfies user-specified requirements on absolute or relative precision as well as coverage probability. ASAP3 operates as follows: the batch size is progressively increased until the batch means pass the Shapiro-Wilk test for multivariate normality; and then ASAP3 fits a first-order autoregressive (AR(1)) time series model to the batch means. If necessary, the batch size is further increased until the autoregressive parameter in the AR(1) model does not significantly exceed 0.8. Next, ASAP3 computes the terms of an inverse Cornish-Fisher expansion for the classical batch means t -ratio based on the AR(1) parameter estimates; and finally ASAP3 delivers a correlation-adjusted confidence interval based on this expansion. Regarding not only conformance to the precision and coverage-probability requirements but also the mean and variance of the half-length of the delivered confidence interval, ASAP3 compared favorably to other batch means procedures (namely, ABATCH, ASAP, ASAP2, and LBATCH) in an extensive experimental performance evaluation.

AutoSimOA: a framework for automated analysis of simulation output

“…Bearing this in mind we selected a method from each category, LABATCH2 (Fishman, 1998) and ASAP3 (Steiger et al, 2005), for implementation within AutoSimOA. Both of these methods are fairly recent and received good recommendations and test results in the literature (Fishman, 1998;Steiger et al, 2005;Alexopoulos, 2006, Lada et al, 2008.…”

“…Both of these methods are fairly recent and received good recommendations and test results in the literature (Fishman, 1998;Steiger et al, 2005;Alexopoulos, 2006, Lada et al, 2008. The LABATCH2 algorithm works on a fixed quantity of data, providing a confidence interval without set precision.…”

“…Mechanic & McKay (Mechanic & McKay, 1966) Law & Carson (Law & Carson,1979) Family B2: ASAP (Steiger & Wilson, 1999) ASAP2 (Steiger et al, 2002) ASAP3 (Steiger et al, 2005) SBATCH (Lada et al, 2008) IBATCH (Yazdi & Wilson, 2007) WASSP (Lada & Wilson, 2006) Family B3:…”

“…Weighted Batch Means (Bischak, Kelton & Pollack, 1993) Quasi-Independent and Normal Procedure (Chen & Kelton, 2007) Sherman's Algorithm (Sherman, 1995) Orphan B6: this expansion (Steiger et al, 2005). Although ASAP3 attempts to deal with any initial bias in the data by discarding the first two batches of data, it was found not to cope well with substantial bias in that it required extremely (and prohibitively) large amounts of data.…”

State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments

“…For steady-state simulations this control includes specifying the warmup period and run length(s), as well as how (if at all) the time-series output is averaged or aggregated into batches. The choice of the number of batches and batch sizes is an important topic of research in itself (e.g., Schmeiser 1982, Steiger et al 2005, Alexopoulos and Goldsman 2004, and an implicit assumption in many simulation-analysis techniques is that appropriate batch sizes and warm-up periods are used. Other simulation-specific factors that can be controlled include the use of CRNs to facilitate comparisons across alternatives.…”