Finite-Sample Performance of Absolute Precision Stopping Rules

Abstract

A bsolute precision stopping rules are often used to determine the length of sequential experiments to estimate confidence intervals for simulated performance measures. Much is known about the asymptotic behavior of such procedures. In this paper, we introduce coverage contours to quantify the trade-offs in interval coverage, stopping times, and precision for finite-sample experiments using absolute precision rules. We use these contours to evaluate the coverage of a basic absolute precision stopping rule, and we show that this rule will lead to a bias in coverage even if all of the assumptions supporting the procedure are true. We define optimal stopping rules that deliver nominal coverage with the smallest expected number of observations. Contrary to previous asymptotic results that suggest decreasing the precision of the rule to approach nominal coverage in the limit, we find that it is optimal to increase the confidence coefficient used in the stopping rule, thus obtaining nominal coverage in a finite-sample experiment. If the simulation data are independent and identically normally distributed, we can calculate coverage contours analytically and find a stopping rule that is insensitive to the variance of the data while delivering at least nominal coverage for any precision value.

Selecting Stopping Rules for Confidence Interval Procedures

“…Schmeiser and Yeh [2002] use coverage functions to develop a single dimensionless criterion by integrating the deviation of the coverage function from nominal coverage. [Schruben 1980] η η * Coverage contours [Singham and Schruben 2012] η, δ…”

“…Coverage contours compare rules over the space of (η, δ) and display metrics such as η * and Ek * [Singham and Schruben 2012]. In the next section, we introduce coverage profiles that map the coverage according to the stopping time of a rule.…”

“…Approaches to evaluating losses in the coverage and choosing optimal parameters for absolute-precision stopping rules were developed in Singham and Schruben [2012]. That paper derived the coverage when the simulation output was independent and identically distributed (i.i.d.)…”

“…Traditionally, small desired precision values have been suggested to push the sample size high enough to obtain a coverage that is close to nominal. We compare this method and new methods with those suggested in Singham and Schruben [2012], and analyze the costs associated with each method.…”

“…We use CIP1 as an example throughout this article because it is simple enough that when applied to i.i.d. normal data, the coverage and distribution of k * can be derived explicitly as in Singham and Schruben [2012], but the methodology and graphical tools developed can apply to many other types of rules.…”