On Using Truncated Sequential Probability Ratio Test Boundaries for Monte Carlo Implementation of Hypothesis Tests

Abstract

When designing programs or software for the implementation of Monte Carlo (MC) hypothesis tests, we can save computation time by using sequential stopping boundaries. Such boundaries imply stopping resampling after relatively few replications if the early replications indicate a very large or very small p-value. We study a truncated sequential probability ratio test (SPRT) boundary and provide a tractable algorithm to implement it. We review two properties desired of any MC p-value, the validity of the p-value and a small resampling risk, where resampling risk is the probability that the accept/reject decision will be different than the decision from complete enumeration. We show how the algorithm can be used to calculate a valid p-value and confidence intervals for any truncated SPRT boundary. We show that a class of SPRT boundaries is minimax with respect to resampling risk and recommend a truncated version of boundaries in that class by comparing their resampling risk (RR) to the RR of fixed boundaries with the same maximum resample size. We study the lack of validity of some simple estimators of p-values and offer a new simple valid p-value for the recommended truncated SPRT boundary. We explore the use of these methods in a practical example and provide the MChtest R package to perform the methods.

Finding Test Statistic Thresholds Using Simulation and Model Fitting with an Application to Radiation Detection

“…First, we apply the SPRT to the count data from each container inspection, which has a predetermined maximum run length determined by the maximum count time; consequently, Wald's thresholds, which are theoretically intended for an infinite sequence of counts, do not apply. A truncated version of SPRT was studied by Fay et al (2007) using Monte Carlo, but the focus was implementation of Monte Carlo hypothesis testing. Second, the background mean count is estimated by a running average of counts that are independent of the container counts; consequently, estimation error in the background mean count must be considered.…”

Bounding the resampling risk for sequential Monte Carlo implementation of hypothesis tests

“…A simple sequential design is a curtailed design where we stop and reject or accept the null hypothesis when we have the replications enough to ensure the rejection or acceptance of the null hypothesis with a full enumeration of the MC test. Fay et al (2007) proposed using a truncated sequential probability ratio (tSPRT) boundary which is minimax with respect to the resampling risk, and provided an algorithm to calculate a valid p-value after reaching the stopping boundary. As discussed in Fay et al (2007), there is another way to determine an MC boundary: by minimizing the resampling risk over a class of possible distributions for the p-value.…”

“…Fay et al (2007) proposed using a truncated sequential probability ratio (tSPRT) boundary which is minimax with respect to the resampling risk, and provided an algorithm to calculate a valid p-value after reaching the stopping boundary. As discussed in Fay et al (2007), there is another way to determine an MC boundary: by minimizing the resampling risk over a class of possible distributions for the p-value. Fay and Follmann (2002) used the class of distributions for the p-values generated by location shifts for a standard normal test statistic and approximated the associated distributions for the p-values within this class using beta distributions.…”

“…Fay et al (2007) did not fully explore using the Fay and Follmann (2002) approach for bounding the resampling risk within a class of distributions for the p-value, but only plotted resampling risk as a function of fixed p-values. In this paper, we explore the Fay and Follmann (2002) approach and focus on sequential designs which bound or approximately bound the resampling risk within the previously mentioned class of distributions for the p-value.…”

“…The B-value is a statistic which produces the expected value of the test statistic at the end of study, conditional on information up to some point, and we make a decision based on this projected value. We construct the B-value boundary either by using the numerical algorithm introduced in Fay et al (2007), FKH algorithm, or by using approximations introduced in this paper. We call the latter design the approximate B-value design, and it is much easier to construct, but it only provides a decision rule, without estimating the p-value.…”