Obtaining evidence for no effect

Dienes, Zoltán

doi:10.31234/osf.io/yc7s5

Cited by 3 publications

(4 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other procedures, such as equivalence testing (Lakens, McLatchie, Isager, Scheel, & Dienes, 2020), could be explored as an alternative to the use of Bayes factors. However, equivalence testing relies on the determination of minimal interesting effect sizes (Dienes, 2020), which are problematic in the context of determination of unawareness. Moreover, Bayes factors have a crucial advantage over equivalence testing, as they allow quantifying evidence for the null hypothesis.…”

Section: Discussionmentioning

confidence: 99%

Avoiding pitfalls: Bayes factors can be a reliable tool for post hoc data selection in implicit learning

Leganes-Fonteneau¹,

Scott²,

Duka³

et al. 2020

Preprint

View full text Add to dashboard Cite

Research on implicit processes has revealed problems with awareness categorizations based on non-significant results. Moreover, post-hoc categorizations result in regression to the mean (RTM), by which aware participants are wrongly categorized as unaware. Using Bayes factors to obtain sensitive evidence for participants’ lack of knowledge may deal with non-significance being non-evidential but also may prevent regression-to-the-mean effects. Here we examine the reliability of a novel Bayesian awareness categorization procedure.Participants completed a reward learning task followed by a Flanker task measuring attention towards conditioned stimuli. They were categorized as B_Aware and B_Unaware of stimulus-outcome contingencies, and those with insensitive Bayes factors were deemed B_Insensitive. We found that performance for B_Unaware participants was below chance level using unbiased tests. This was further confirmed using a resampling procedure with multiple iterations, contrary to the prediction of RTM effects. Conversely, when categorizing participants using t-tests, t_Unaware participants showed RTM effects. We also propose a group boundary optimization procedure to determine the threshold at which regression to the mean is observed. Using Bayes factors instead of t-tests as a post-hoc categorization tool allows evaluating evidence of unawareness, which in turn helps avoid RTM. The reliability of the Bayesian awareness categorization procedure strengthens previous evidence for implicit reward conditioning. The toolbox used for the categorization procedure is detailed and made available. Post-hoc group selection can provide evidence for implicit processes; the relevance of RTM needs to be considered for each study and cannot simply be assumed to be a problem.

show abstract

Section: Discussionmentioning

confidence: 99%

Avoiding pitfalls: Bayes factors can be a reliable tool for post hoc data selection in implicit learning

Leganes-Fonteneau¹,

Scott²,

Duka³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In general, calibration is the process of converting from one scale about which it is hard to judge what is of interest, to another scale for which, as it happens, we can judge. See Dienes (2020) for other examples and heuristics.…”

Section: Analytic Flexibility Being Tied Down While Ensuring Sensitivmentioning

confidence: 99%

“…That is, the effect size used to scale the predictions of the model of H1 must be the rough effect size predicted by the theory. Dienes (2019Dienes ( , 2020 provide heuristics for determining what rough effect size a theory predicts. This may involve collecting some pilot data, and not necessarily for the full design.…”

Section: Analytic Flexibility Being Tied Down While Ensuring Sensitivmentioning

confidence: 99%

“…Note the positive coefficients sum to 1 and the negative ones to -1; the expected size of L in raw units is just a bit smaller than the expected difference between the two extreme conditions. Thus, the heuristics discussed in Dienes (2019Dienes ( , 2020 for the difference between conditions may be useful for such contrasts. (Useful rule of thumb: In calculating a contrast, make positive coefficients sum to 1, negative to -1, then it may be easy to think about expected effect sizes for the contrast in raw units.…”

Section: Lining Up Predictions and Testsmentioning

confidence: 99%

See 1 more Smart Citation

The inner workings of Registered Reports

Dienes¹

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Registered Reports provide one way to address shortcomings in the current way we manage research, right from the design of studies to their publication. The format requires pre-specifying why a design may crucially test a theory, what auxiliary assumptions are required for the experiment to be such a test, what outcome neutral tests are required in turn to test those assumptions, what specific crucial tests will therefore be used test the theory (of the many tests that could be used), and why those tests could provide evidence for no effect of interest given the proposed numbers of trials and participants. Reviewers and authors are then constructively involved in optimizing the experiment before it is run. The agreement between reviewers and authors, as adjudicated by the editors, then defines in advance the proposed method and analytic protocol, virtually guaranteeing acceptance of the paper, no matter what position, if any, the results support. Here we go through what problems the format solves, and why it must therefore be approached in a way that is little understood. Common errors are discussed. The paper thus provides an argument for how to approach Registered Reports for readers, authors, and editors of the format.

show abstract