Improved batching for confidence interval construction in steady-state simulation

Abstract

We describe an improved batch-means procedure for building a confidence interval on a steady-state expected simulation response that is centered on the sample mean of a portion of the corresponding simulation-generated time series and satisfies a user-specified absolute or relative precision requirement. The theory supporting the new algorithm merely requires the output process to be weakly dependent (phi-mixing) so that for a sufficiently large batch size, the batch means are approximately multivariate normal but not necessarily uncorrelated. A variant of the method of nonoverlapping batch means (NOBM), the Automated Simulation Analysis Procedure (ASAP) operates as follows: the batch size is progressively increased until either (a) the batch means pass the von Neumann test for independence, and then ASAP delivers a classical NOBM confidence interval; or (b) the batch means pass the Shapiro-Wilk test for multivariate normality, and then ASAP delivers a corrected confidence interval. The latter correction is based on an inverted Cornish-Fisher expansion for the classical NOBM t-ratio, where the terms of the expansion are estimated via an autoregressive-moving average time series model of the batch means. An experimental performance evaluation demonstrates the advantages of ASAP versus other widely used batch-means procedures.

An Improved Batch Means Procedure for Simulation Output Analysis

“…Although normality of each four-dimensional random vector y is not sufficient to ensure joint normality of all k 1 = 94 batch means (Stuart and Ord 1994, Exercise 15.20), our computational experience strongly suggests that this approach to testing for joint normality of the batch means is effective in practical applications of ASAP (Steiger 1999). Thus, another reason for the choice of 96 initial batches is that it yields a reasonably sensitive test for multivariate normality of the batch means on each iteration of ASAP requiring such a test.…”

“…If the batch means fail the normality test, then the batch size is increased by the factor √ 2, and the entire process is repeated until the batch means finally pass one of the two tests. In our computational experience (Steiger 1999, Steiger andWilson 2001), the normality test has greater sensitivity to increases in the batch size than the independence test has. Thus, to construct usable batch means ASAP does not require doubling the batch size on each iteration as required, for example, on those iterations of the simulation analysis procedures LBATCH and ABATCH for which the batch means fail the independence test (Fishman andYarberry 1997, Fishman 1998).…”

“…Finally, in §8 we recapitulate our main findings, and we make recommendations for future work. Although this paper is based on Steiger (1999), some preliminary results on the formulation of ASAP and its experimental performance evaluation were presented in Wilson (1999, 2000).…”

“…During preliminary experimentation with ASAP on the same six processes that were used in our investigation of the convergence properties of batch means (Steiger 1999, Steiger andWilson 2001), we found that ASAP delivered the best overall results in terms of confidence-interval coverage probability when we set ind = 0 20 and mvn = 0 10. Subsequently, we used these values in the comprehensive experimental performance evaluation of ASAP that is described in §7.…”

“…Then the "best" fit of the five models is chosen. Preference is given to the AR(1) model (7) because in all our computational experience with ASAP, the AR(1) model has consistently produced superior fits to simulation-generated sequences of batch means (Steiger 1999). If the MA(1) model has a residual variance less than or equal to 75% of the residual variance from the AR(1) model, then it is chosen as the "best-fitting" first-order model.…”