Avoiding pitfalls: Bayes factors can be a reliable tool for post hoc data selection in implicit learning

Abstract

Research on implicit processes has revealed problems with awareness categorizations based on non-significant results. Moreover, post-hoc categorizations result in regression to the mean (RTM), by which aware participants are wrongly categorized as unaware. Using Bayes factors to obtain sensitive evidence for participants’ lack of knowledge may deal with non-significance being non-evidential but also may prevent regression-to-the-mean effects. Here we examine the reliability of a novel Bayesian awareness categorization procedure.Participants completed a reward learning task followed by a Flanker task measuring attention towards conditioned stimuli. They were categorized as B_Aware and B_Unaware of stimulus-outcome contingencies, and those with insensitive Bayes factors were deemed B_Insensitive. We found that performance for B_Unaware participants was below chance level using unbiased tests. This was further confirmed using a resampling procedure with multiple iterations, contrary to the prediction of RTM effects. Conversely, when categorizing participants using t-tests, t_Unaware participants showed RTM effects. We also propose a group boundary optimization procedure to determine the threshold at which regression to the mean is observed. Using Bayes factors instead of t-tests as a post-hoc categorization tool allows evaluating evidence of unawareness, which in turn helps avoid RTM. The reliability of the Bayesian awareness categorization procedure strengthens previous evidence for implicit reward conditioning. The toolbox used for the categorization procedure is detailed and made available. Post-hoc group selection can provide evidence for implicit processes; the relevance of RTM needs to be considered for each study and cannot simply be assumed to be a problem.

How (not) to demonstrate unconscious priming: Overcoming issues with post-hoc data selection, low power, and frequentist statistics

“…As Schmidt (2015) succinctly put it: "Selecting only those participants […] that meet a specified visibility criterion is analogous to testing a new medication and then discarding all those patients who die from it, concluding that all "suitable" patients do fine under the new drug." One possible solution is to employ independent datasets for selecting and testing the potentially unconscious sub-group of participants (Shanks, 2017; for alternative approaches, see Leganes-Fonteneau et al, 2021;Rothkirch et al, 2022;Yaron et al, 2023). Following Schmidt's (2015) analogy, patients could be screened for eligibility in a trial testing a new medication, allowing only suitable patients to enter the trial; whether these patients then do fine (or die) under the new drug will be evaluated in a next step, using data independent from the screener.…”