Decisions About Equivalence: A Comparison of TOST, HDI-ROPE, and the Bayes Factor

Linde, Maximilian; Tendeiro, Jorge N.; Selker, Ravi; Wagenmakers, Eric‐Jan; Ravenzwaaij, Don van

doi:10.31234/osf.io/bh8vu

Cited by 14 publications

(24 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…35]. However, recent simulation studies have also demonstrated that equivalence tests for replications can be either too liberal (especially in the case of publication bias) [ 68 ], or too conservative (especially in case of low sample sizes) [ 69 ].…”

Section: Methodsmentioning

confidence: 99%

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

et al. 2022

View full text Add to dashboard Cite

The attentional blink (AB) phenomenon reveals a bottleneck of human information processing: the second of two targets is often missed when they are presented in rapid succession among distractors. In our previous work, we showed that the size of the AB can be changed by applying transcranial direct current stimulation (tDCS) over the left dorsolateral prefrontal cortex (lDLPFC) (London & Slagter, Journal of Cognitive Neuroscience, 33, 756–68, 2021). Although AB size at the group level remained unchanged, the effects of anodal and cathodal tDCS were negatively correlated: if a given individual’s AB size decreased from baseline during anodal tDCS, their AB size would increase during cathodal tDCS, and vice versa. Here, we attempted to replicate this finding. We found no group effects of tDCS, as in the original study, but we no longer found a significant negative correlation. We present a series of statistical measures of replication success, all of which confirm that both studies are not in agreement. First, the correlation here is significantly smaller than a conservative estimate of the original correlation. Second, the difference between the correlations is greater than expected due to sampling error, and our data are more consistent with a zero-effect than with the original estimate. Finally, the overall effect when combining both studies is small and not significant. Our findings thus indicate that the effects of lDPLFC-tDCS on the AB are less substantial than observed in our initial study. Although this should be quite a common scenario, null findings can be difficult to interpret and are still under-represented in the brain stimulation and cognitive neuroscience literatures. An important auxiliary goal of this paper is therefore to provide a tutorial for other researchers, to maximize the evidential value from null findings.

show abstract

Section: Methodsmentioning

confidence: 99%

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

et al. 2022

View full text Add to dashboard Cite

show abstract

“…They will sometimes therefore give apparently different answers. For example, Dienes (2016) and Linde et al (2020) found that inference by intervals is often unable to give a definitive answer when Bayes factors can provide support for the null model. The Bayes factor in modelling H1 as a probability distribution, uses more information about the theory (that predicts a difference), and hence can often draw more definite conclusions, where there is relevant information to be used.…”

Section: Discussionmentioning

confidence: 99%

Obtaining Evidence for No Effect

Dienes

2021

Collabra: Psychology

View full text Add to dashboard Cite

Obtaining evidence that something does not exist requires knowing how big it would be were it to exist. Testing a theory that predicts an effect thus entails specifying the range of effect sizes consistent with the theory, in order to know when the evidence counts against the theory. Indeed, a theoretically relevant effect size must be specified for power calculations, equivalence testing, and Bayes factors in order that the inferential statistics test the theory. Specifying relevant effect sizes for power, or the equivalence region for equivalence testing, or the scale factor for Bayes factors, is necessary for many journal formats, such as registered reports, and should be necessary for all articles that use hypothesis testing. Yet there is little systematic advice on how to approach this problem. This article offers some principles and practical advice for specifying theoretically relevant effect sizes for hypothesis testing.

show abstract

“…Camerer et al, 2018). However, recent simulation studies have also demonstrated that equivalence tests for replications can be either too liberal (especially in the case of publication bias) (Muradchanian, Hoekstra, Kiers, & Ravenzwaaij, 2020), or too conservative (especially in case of low sample sizes) (Linde, Tendeiro, Selker, Wagenmakers, & Ravenzwaaij, 2020).…”

Section: Methodsmentioning

confidence: 99%

“…32]. However, recent simulation studies have also demonstrated that equivalence tests for replications can be either too liberal (especially in the case of publication bias) [67], or too conservative (especially in case of low sample sizes) [68].…”

Section: This Is Known As An Inferiority Test [34]mentioning

confidence: 99%

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

Reteig

Newman

Ridderinkhof

et al. 2021

Preprint

View full text Add to dashboard Cite

The attentional blink (AB) phenomenon reveals a bottleneck of human information processing: the second of two targets is often missed when they are presented in rapid succession among distractors. A recent study by London & Slagter (Journal of Cognitive Neuroscience, 33, 756 to 768, 2021) showed that the size of the AB can be changed by applying transcranial direct current stimulation (tDCS) over the left dorsolateral prefrontal cortex (lDLPFC). Although AB size at the group level remained unchanged, the effects of anodal and cathodal tDCS were negatively correlated: if a given individuals AB size decreased from baseline during anodal tDCS, their AB size would increase during cathodal tDCS, and vice versa. Here, we attempted to replicate this finding. Like London & Slagter, we found no group effects of tDCS, but also no longer found a significant negative correlation. We present a series of statistical measures of replication success, all of which confirm that both studies are not in agreement. First, the correlation here is significantly smaller than a conservative estimate of the original correlation. Second, the difference between the correlations is greater than expected due to sampling error, and our data are more consistent with a zero effect than with the original estimate. Finally, the overall effect when combining both studies is small and not significant. Our findings thus indicate that the effects of lDPLFC tDCS on the AB are less substantial than observed by London & Slagter (2021). Although this should be quite a common scenario, null findings can be difficult to interpret and are still under represented in the brain stimulation and cognitive neuroscience literatures. An important auxiliary goal of this paper is therefore to provide a tutorial for other researchers, to maximize the evidential value from null findings.

show abstract

Decisions About Equivalence: A Comparison of TOST, HDI-ROPE, and the Bayes Factor

Cited by 14 publications

References 58 publications

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

Obtaining Evidence for No Effect

Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt

Contact Info

Product

Resources

About