Performance evaluation of recent procedures for steady-state simulation analysis

Abstract

The performance of the batch-means procedure ASAP3 and the spectral procedure WASSP is evaluated on test problems with characteristics typical of practical applications of steady-state simulation analysis procedures. ASAP3 and WASSP are sequential procedures designed to produce a confidence-interval estimator for the mean response that satisfies user-specified half-length and coverage-probability requirements. ASAP3 is based on an inverse Cornish-Fisher expansion for the classical batch-means t-ratio, whereas WASSP is based on a wavelet estimator of the batch-means power spectrum. Regarding closeness of the empirical coverage probability and average half-length of the delivered confidence intervals to their respective nominal levels, both procedures compared favorably with the Law-Carson procedure and the original ASAP algorithm. Regarding the average sample sizes required for decreasing levels of maximum confidence-interval half-length, ASAP3 and WASSP exhibited reasonable efficiency in the test problems.

N-Skart: A Nonsequential Skewness- and Autoregression-Adjusted Batch-Means Procedure for Simulation Analysis

“…The performance of ABATCH as reported in Table II is consistent with the following: 1) the results reported in [2,Figs. 2 and 4]; and 2) all our computational experience with ABATCH as reported in [3], [7]- [9]. On the other hand, we found that N-Skart outperformed ABATCH for all four sample sizes considered in Table II; and for , we found that the coverage probabilities for the CIs delivered by N-Skart did not deviate significantly from their nominal levels with respect to either statistical or practical significance.…”

“…N-Skart then computes the latest values of the associated grand mean and sample variance of the truncated, nonspaced batch means according to (3).…”

A Comprehensive Review of Methods for Simulation Output Analysis

“…Related papers, experimental results, and the ASAP3 software are accessible from the site <http://www.ie.ncsu.edu/jwilson>. A detailed experimental study of sequential NBM procedures is presented in Lada et al (2006).…”

Skart: A skewness- and autoregression-adjusted batch-means procedure for simulation analysis

“…From Table 2 of Steiger and Wilson (2002) for example, ABATCH delivered the following coverage probabilities for nominal 90% CIs with the indicated relative precision levels: (i) no precision, 60%; (ii)˙15% precision, 72%; and (iii) 7:5% precision, 82%. From Table 4 of Lada, Steiger, and Wilson (2006), the corresponding coverage probabilities for the Law-Carson procedure are 85%, 85%, and 87%, respectively. From Table 2 of Lada et al (2007), the corresponding coverage probabilities for the Heidelberger-Welch procedure are 67.8%, 76%, and 77%.…”

“…There is substantial experimental evidence that ASAP3 outperforms ABATCH and LBATCH (Steiger and Wilson 2002) and the Law-Carson procedure (Lada, Steiger, and Wilson 2006). Similarly Lada et al (2007) provide good experimental evidence that WASSP outperforms the Heidelberger-Welch procedure.…”