Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test

Abstract

In a sequential hypothesis test, the analyst checks at multiple steps during data collection whether sufficient evidence has accrued to make a decision about the tested hypotheses. As soon as sufficient information has been obtained, data collection is terminated. Here, we compare two sequential hypothesis testing procedures that have recently been proposed for use in psychological research: Sequential Probability Ratio Test (SPRT; Psychological Methods, 25(2), 206–226, 2020) and the Sequential Bayes Factor Test (SBFT; Psychological Methods, 22(2), 322–339, 2017). We show that although the two methods have different philosophical roots, they share many similarities and can even be mathematically regarded as two instances of an overarching hypothesis testing framework. We demonstrate that the two methods use the same mechanisms for evidence monitoring and error control, and that differences in efficiency between the methods depend on the exact specification of the statistical models involved, as well as on the population truth. Our simulations indicate that when deciding on a sequential design within a unified sequential testing framework, researchers need to balance the needs of test efficiency, robustness against model misspecification, and appropriate uncertainty quantification. We provide guidance for navigating these design decisions based on individual preferences and simulation-based design analyses.

Waldian t tests: Sequential Bayesian t tests with controlled error probabilities.

“…Optional stopping is thus not a problem in Bayesian t tests (but see de Heide & Grünwald, 2021). In fact, sequential Bayes factors have been proposed repeatedly as a means to increase efficiency in hypothesis testing (Rouder, 2014; Schönbrodt et al, 2017; Stefan et al, 2022; Wagenmakers et al, 2012).…”

“…Although threshold values for Bayes factors can in principle be calibrated to have the desired error probabilities under the specified statistical models, this calibration may not be straightforward except for atypical applications where both scriptH0 and scriptH1 correspond to simple point hypotheses (see Frequentist Error Probabilities in Bayesian t Tests section). In typical applications with more complex specifications of scriptH1, error probability control for Bayes factors will involve sophisticated calculations and potentially time-consuming simulations (Schönbrodt et al, 2017; Stefan et al, 2022).…”

“…Recent simulation studies have demonstrated that they efficiently control the error probabilities of statistical decisions (Schnuerch & Erdfelder, 2020). Importantly, the likelihood ratio underlying these sequential t tests is closely related to the Bayes factor in Bayesian t tests (Stefan et al, 2022). Building on this relationship, we show that Rushton’s and Hajnal’s sequential t tests can be generalized beyond the case of point hypotheses to a more general class of alternative hypotheses as represented by prior distributions.…”

“…For example, Berger and Bayarri (2004, p. 58) argue that “statisticians should readily use both Bayesian and frequentist ideas”. There have been previous efforts in the literature to estimate error probabilities of Bayesian t tests by simulation (e.g., Jeon & De Boeck, 2017; Sanborn & Hills, 2014; Schnuerch & Erdfelder, 2020; Schönbrodt et al, 2017; Yu et al, 2014), to control them explicitly (e.g., Gu et al, 2016; Hoijtink et al, 2016; Schönbrodt & Wagenmakers, 2018; Stefan et al, 2022), and to unify Bayesian and frequentist test procedures (e.g., Bayarri et al, 2016; Berger, 2003; Berger et al, 1997, 1999). In contrast to the frequentist error probabilities that we consider in this article, however, such unification efforts have focused mostly on error rates conditional on the observed data (Berger et al, 1994).…”

“…In fact, because LR N almost always strictly exceeds the boundary at termination (a phenomenon called overshooting ), the resulting error rates of the procedure will undercut the nominal α and β. If exact error control is required, the threshold values A and B can be numerically adjusted to result in the nominal error rates for the specified hypotheses (Stefan et al, 2022). However, Wald (1947) conjectured that the decrease in the SPRT’s efficiency due to its conservative behavior is negligible, and recent simulation studies have corroborated this supposition (Schnuerch et al, 2020; Schnuerch & Erdfelder, 2020).…”