N-Skart: A Nonsequential Skewness- and Autoregression-Adjusted Batch-Means Procedure for Simulation Analysis

Tafazzoli, Ali; Steiger, Natalie M.; Wilson, James R.

doi:10.1109/tac.2010.2052137

Cited by 22 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Stopping rules are typically incorporated in CIPs that seek to return valid confidence intervals for data that may be dependent or nonnormal (Chen and Kelton 2007, Hoad et al 2009, Steiger and Wilson 2002, Tafazzoli et al 2011. Sequential ad hoc stopping rules are sometimes tailored to a specific set of simulation test models to provide better results.…”

Section: Introductionmentioning

confidence: 99%

Finite-Sample Performance of Absolute Precision Stopping Rules

Singham

Schruben

2012

INFORMS Journal on Computing

View full text Add to dashboard Cite

A bsolute precision stopping rules are often used to determine the length of sequential experiments to estimate confidence intervals for simulated performance measures. Much is known about the asymptotic behavior of such procedures. In this paper, we introduce coverage contours to quantify the trade-offs in interval coverage, stopping times, and precision for finite-sample experiments using absolute precision rules. We use these contours to evaluate the coverage of a basic absolute precision stopping rule, and we show that this rule will lead to a bias in coverage even if all of the assumptions supporting the procedure are true. We define optimal stopping rules that deliver nominal coverage with the smallest expected number of observations. Contrary to previous asymptotic results that suggest decreasing the precision of the rule to approach nominal coverage in the limit, we find that it is optimal to increase the confidence coefficient used in the stopping rule, thus obtaining nominal coverage in a finite-sample experiment. If the simulation data are independent and identically normally distributed, we can calculate coverage contours analytically and find a stopping rule that is insensitive to the variance of the data while delivering at least nominal coverage for any precision value.

show abstract

Section: Introductionmentioning

confidence: 99%

Finite-Sample Performance of Absolute Precision Stopping Rules

Singham

Schruben

2012

INFORMS Journal on Computing

View full text Add to dashboard Cite

show abstract

“…The experimental results show that Skart compares favorably with other steady-state simulation analysis procedures-namely, its predecessors WASSP Lada et al, 2007), ASAP3, and SBatch (Lada et al, 2008), as well as sequential versions of LABATCH.2, the procedure of Law and Carson (1979), and the spectral procedure of Heidelberger and Welch (1981). Tafazzoli et al (2011a) also developed N-Skart, a nonsequential version of Skart that works with a fixedsize data set. In an experimental performance evaluation involving the same test processes as for Skart, the authors find that N-Skart outperforms LABATCH.2.…”

Section: Overview Of the Skart Proceduresmentioning

confidence: 98%

SPSTS: A sequential procedure for estimating the steady-state mean using standardized time series

et al. 2016

Self Cite

View full text Add to dashboard Cite

This article presents SPSTS, an automated sequential procedure for computing point and confidence-interval (CI) estimators for the steady-state mean of a simulation-generated process subject to user-specified requirements for the CI coverage probability and relative half-length. SPSTS is the first sequential method based on standardized time series (STS) area estimators of the steady-state variance parameter (i.e., the sum of covariances at all lags). Whereas its leading competitors rely on the method of batch means to remove bias due to the initial transient, estimate the variance parameter, and compute the CI, SPSTS relies on the signed areas corresponding to two orthonormal STS area variance estimators for these tasks. In successive stages of SPSTS, standard tests for normality and independence are applied to the signed areas to determine (i) the length of the warm-up period; and (ii) a batch size sufficient to ensure adequate convergence of the associated STS area variance estimators to their limiting chi-squared distributions. SPSTS's performance is compared experimentally with that of recent batch-means methods using selected test problems of varying degrees of difficulty. SPSTS performed comparatively well in terms of its average required sample size as well as the coverage and average half-length of the final CIs. ARTICLE HISTORYKEYWORDS Steady-state simulation; simulation output analysis; method of batch means; method of standardized time series; area variance estimator; nonoverlapping variance estimator; overlapping variance estimator uiie-3995r1-final.tex 1

show abstract

“…If we suspect our data are non-normal, we can use a modified Cornish-Fisher expansion as is done in Tafazzoli et al [2011] to obtain confidence intervals that are calculated using the skewness of the data. In this case, one of the statistics in the vector v k would be the sample skewness.…”

Section: Generalized Stopping Rulesmentioning

confidence: 99%

“…However, in many cases, the nature of the distribution and dependence will be unknown before the experiment begins. One strategy, presented by Tafazzoli et al [2011], is to estimate the dependence and deviation from normality as batches of data are collected and to adjust the t-value used for the confidence intervals accordingly. The authors are able to achieve an improved coverage for many scenarios involving nonnormal distributions by using a Cornish-Fisher adjustment suggested by Johnson [1978], and then using the von Neumann test for randomness to determine when batch means are approximately independent.…”

Section: Adjusting For the Sample Skewnessmentioning

confidence: 99%

“…We use the adjustment for the skewness of the data as implemented by Tafazzoli et al [2011] as part of CIP1 applied to different data types. Table III lists the results for two stopping rules, (0.90, 0.30, 2) and (0.95, 0.15, 2), where the second rule requires intervals with a higher confidence coefficient and a smaller precision.…”

Section: Adjusting For the Sample Skewnessmentioning

confidence: 99%

See 1 more Smart Citation

Selecting Stopping Rules for Confidence Interval Procedures

Singham

2014

ACM Trans. Model. Comput. Simul.

View full text Add to dashboard Cite

The sample size decision is crucial to the success of any sampling experiment. More samples imply better confidence and precision in the results, but require higher costs in terms of time, computing power, and money. Analysts often choose sequential stopping rules on an ad hoc basis to obtain confidence intervals with desired properties without requiring large sample sizes. However, the choice of stopping rule can affect the quality of the interval produced in terms of the coverage, precision, and replication cost. This article introduces methods for choosing and evaluating stopping rules for confidence interval procedures. We develop a general framework for assessing the quality of a broad class of stopping rules applied to independent and identically distributed data. We introduce coverage profiles that plot the coverage according to the stopping time and reveal situations when the coverage could be unexpectedly low. Finally, we recommend simple techniques for obtaining acceptable or optimal rules.

show abstract

N-Skart: A Nonsequential Skewness- and Autoregression-Adjusted Batch-Means Procedure for Simulation Analysis

Cited by 22 publications

References 15 publications

Finite-Sample Performance of Absolute Precision Stopping Rules

Finite-Sample Performance of Absolute Precision Stopping Rules

SPSTS: A sequential procedure for estimating the steady-state mean using standardized time series

Selecting Stopping Rules for Confidence Interval Procedures

Contact Info

Product

Resources

About