On the reliability of value-modulated attentional capture: An online replication and multiverse analysis

Abstract

Stimuli predicting rewards are more likely to capture attention, even when they are not relevant to our current goals. Individual differences in value-modulated attentional capture (VMAC) have been associated with various psychopathological conditions in the scientific literature. However, the claim that this attentional bias can predict individual differences requires further exploration of the psychometric properties of the most common experimental paradigms. The current study replicated the VMAC effect in a large online sample (N = 182) and investigated the internal consistency, with a design that allowed us to measure the effect during learning (rewarded phase) and after acquisition, once feedback was omitted (unrewarded phase). Through the rewarded phase there was gradual increase of the VMAC effect, which did not decline significantly throughout the unrewarded phase. Furthermore, we conducted a reliability multiverse analysis for 288 different data preprocessing specifications across both phases. Specifications including more blocks in the analysis led to better reliability estimates in both phases, while specifications that removed more outliers also improved reliability, suggesting that specifications with more, but less noisy, trials led to better reliability estimates. Nevertheless, in most instances, especially those considering fewer blocks of trials, reliability estimates fell below the minimum recommended thresholds for research on individual differences. Given the present results, we encourage researchers working on VMAC to take into account reliability when designing studies aimed at capturing individual differences and provide recommendations to improve methodological practices.

Reliably Measuring Learning-Dependent Distractor Suppression with Eye Tracking

“…Using the probability of fixating the distractor, reliable learning-dependent reductions in distractor processing can be observed (Grégoire et al, 2022; Kim & Anderson, 2022), in addition to a measure of attention capture that is reliable for both young and older adults regardless of whether capture is overall suppressed under conditions of feature-search vs. singleton-search. Even when accounting for the increased variance in difference score calculations (Miller & Ulrich, 2013; Paap & Sawi, 2016; Weichselbaum et al, 2018), we demonstrate that oculomotor measures of attention capture on average exhibit strong reliability (mean across acquired values, r = .735) and are considerably more reliable than response time-based measures (Anderson & Kim, 2019; Freichel et al, 2023; Garre-Frutos et al, 2024; Ivanov et al, 2023).…”

“…Instead of utilizing an arbitrary odd vs. even split, we estimated internal consistency by utilizing a permuted random split procedure as in Garre-Frutos et al (2024). In this procedure, all trials were randomly split into two halves with an equal number of observations in each half per condition per run to account for time-dependent effects (e.g., learning or extinction).…”

“…Over the three measures, the authors report poor to moderate split-half reliability over response times and poor reliability over accuracy, in addition to poor test-retest reliability with respect to both response times and accuracy. Furthermore, three studies investigating selection history effects of reward learning in visual search also reported poor test-retest reliability of behavioral response times (Anderson & Kim, 2019; Freichel et al, 2023; Garre-Frutos et al, 2024). These studies collectively identified that response time exhibits poor reliability over experience-driven attention effects.…”

Measurement and sampling noise undermine inferences about awareness in location probability learning: A modeling approach

“…A clear example consists in submitting variables to analyses that implicitly assume they have been measured with little or no measurement noise (Vadillo et al, 2020(Vadillo et al, , 2022. On the occasions when this assumption has been evaluated, the psychometric properties of the measures provided by these tasks often leave much to be desired (e.g., Draheim et al, 2019;Enkavi et al, 2019;Garre-Frutos et al, 2024;Hedge et al, 2018;Paap & Sawi, 2016;Rothkirch et al, 2022;Vadillo et al, 2022;Yaron et al, 2024).…”

Inhibition of Return and Learned Value

“…For example, this specific association appears to alter the ability of that stimulus feature to capture attention (Anderson et al, 2011;Della Libera & Chelazzi, 2009), even when the stimulus predicting reward acts as an irrelevant distractor so that attending to it is completely counterproductive (Le Pelley et al, 2015;Le Pelley et al, 2017), or when reward is no longer available to motivate performance (Anderson & Halpern, 2017;Anderson et al, 2011;Watson et al, 2019;Garre-Frutos et al, 2024). Thus, this bias does not consider endogenous factors or the intrinsic capacity of the stimulus to capture attention and has therefore been proposed as an effect belonging to the so-called selection history (Anderson et al, 2021) that involuntarily influences attentional orienting.…”