Waldian t tests: Sequential Bayesian t tests with controlled error probabilities

Abstract

Bayesian t tests have become an increasingly popular alternative to null-hypothesis significance testing (NHST) in psychological research. In contrast to NHST, they allow for the quantification of evidence in favor of the null hypothesis and for optional stopping. A major drawback of Bayesian t tests, however, is that error probabilities of statistical decisions remain uncontrolled. Previous approaches in the literature to remedy this problem either include time-consuming simulations or the specification of prior distributions without a substantive meaning. In this article, we propose a sequential probability ratio test that combines Bayesian t tests with simple decision criteria developed by Abraham Wald in 1947. We discuss this sequential procedure, which we call Waldian t test, in the context of default and informed Bayesian t tests. We show that Waldian t tests reliably control frequentist error probabilities, with the nominal Type I and Type II error probabilities serving as upper bounds to the actual error rates. At the same time, the prior distributions of Bayesian t tests are preserved. Thus, Waldian t tests are fully justified from both a frequentist and a Bayesian point of view. We highlight the relationship between frequentist and Bayesian error probabilities and critically discuss the implications of conventional stopping criteria for sequential Bayesian t tests. Finally, we provide a user-friendly web application which implements the proposed procedure for substantive researchers.

On the efficiency of the independent segments procedure: A direct comparison with sequential probability ratio tests.

“…Consequently, the nominal α and β values serve as upper bounds to the actual error rates in the SPRT and rarely match them exactly (cf. Schnuerch & Erdfelder, 2020; Schnuerch et al, 2021).…”

“…Given that most of the tests conducted in psychology refer to composite hypotheses, this is a potential limitation. Fortunately, techniques have become available to cope with this limitation in many situations, for example, replacing composite hypotheses by simple hypotheses on transformations of the data (Cox, 1952; Rushton, 1950; Schnuerch & Erdfelder, 2020), using weight functions and integrating out nuisance parameters (Wald, 1947; see also Schnuerch et al, 2021), or replacing unknown nuisance parameters by their maximum likelihood estimates for a given set of data—an asymptotic method that works nicely when initial sample sizes in the SPRT are not too small (Cox, 1963; Schnuerch et al, 2020).…”

The role of domain-general attention and domain-specific processing in working memory in algebraic performance: An experimental approach.

“…For instance, a BF of 10 means that the likelihood of the alternative hypothesis is 10 times the likelihood of the null hypothesis based on the present data (Wetzels & Wagenmakers, 2012). There are different views on how to choose thresholds and name the intervals between them (e.g., Jeffreys, 1961; Kass & Raftery, 1995; Schönbrodt & Stefan, 2018; Schönbrodt et al, 2017; Schnuerch et al, 2021). In the current study, the criteria for the magnitude of the effects were based on a combination of terms for thresholds proposed by others, including Jeffreys (1961), Lee and Wagenmakers (2014), Schönbrodt and Stefan (2018), and van Doorn et al (2021).…”

Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test

“…Schnuerch et al (2021) make a point that the weighted average of all type II error rates based on the effect sizes specified in the prior distribution under M 1 will be smaller than the maximum error rate. However, this does not imply that the effective error rate is smaller than the specified error rate for any specific effect size.…”