Bayesian t-Test Sample Size Determination: Reference Tables for Various Bayes Factor Thresholds, Effect Sizes, Sample Sizes, and Variance Assumptions

Abstract

Psychologists and other social scientists are often interested in statistically testing for differences between two means, with the use of a t¬-test. Recently, Bayesian methods for statistical testing have become more popular amongst psychologists. Bayesian methods, including the Bayeisan t-test result in a ratio, called a Bayes Factor (BF), which indicates the likelihood of one model over another (commonly the alternative vs the null hypothesis models). The BF is an updatable ratio, which increases in favor of the most probable model as data is accumulated. Therefore, the BF enables researchers to decide whether evidence is compelling enough to stop with data collection, or if more observations should be obtained. However, data collection cannot always rely on this updatable property of the BF and sample size determination is often required a-priori. Determining the required sample size within the Bayesian framework, depends upon simulations, which can be timely and complicated to implement. Therefore, we conducted 1600 simulations with 10000 iterations each, to provide sample size determination heuristics for researchers and other stakeholders based on common assumptions expected within the social sciences. We summarize these heuristics in eight reference tables, which provide the probability of reaching five different BF thresholds for a Bayesian t-test, for evidence in favor of both the alternative and the null hypotheses, for 10 possible effect sizes, eight sample sizes, and four different data variability assumptions.

Causal evidence for the role of the sensory visual cortex in visual short-term memory maintenance

“…[ 60 ]). Here, to have an adequate sample size for our registered analyses (see [ 116 ]), we opted for the use of a change-detection task, which was deemed as an appropriate approach (e.g. [ 117 ]) to understand our main research question of whether the sensory visual cortex contributes to VSTM maintenance.…”

“…figure 3 c ). As such, future studies using Bayesian designs could rely on different approaches for sample size determination, such as simulations that can inform the minimum required number of participants for various study designs [ 116 , 120 , 122 ]. Alternatively, future work could set two different BF thresholds for the stopping rule and the decision rule, 2 to allow for additional sensitivity in case exploratory analyses are anticipated.…”