Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks

Kucina, Talira; Wells, Lindsay; Lewis, Ian; Salas, Kristy de; Kohl, Amelia; Palmer, Matthew A.; Sauer, James D.; Matzke, Dora; Aidman, Eugene; Heathcote, Andrew

doi:10.1038/s41467-023-37777-2

Cited by 10 publications

(6 citation statements)

References 59 publications

(71 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first conclusion was at the methodological level. It was proposed that the typical tasks or measures should be abandoned, and new ways of measuring attentional control should be developed (e.g., Burgoyne et al, 2023;Draheim et al, 2021Draheim et al, , 2024Kucina et al, 2023;Martin et al, 2020;Rey-Mermet & Rothen, 2023a, 2023b. In contrast, the second conclusion was at the conceptual level.…”

Section: Beyond the Methodological Challengesmentioning

confidence: 99%

Beyond Unity and Diversity: Tasks Used to Assess Attentional Control Measure Task-Specific Processes

Rey-Mermet

2024

Preprint

View full text Add to dashboard Cite

Attentional control – also referred to as cognitive control or executive functions – is defined as our ability to supervise and control our thoughts and actions in order to achieve our goals. So far, attentional control has been conceptualized either as a unitary construct or as a heterogeneous construct with diverse (sub-)constructs. Irrespective of the conceptualizations, one common assumption is that attentional control – or its (sub-)constructs – can be measured with different tasks. That is, it is not necessary that all tasks measure the same attentional-control process, but at least two different tasks should measure the same attentional-control process. However, recent research has questioned this assumption using three different lines of research. The first line used an experimental approach, the second investigated the impact of age, and the third used a correlational approach. Until now, these different lines of research have been considered apart. In this review, I integrate them, coming the conclusion that the tasks used to assess attentional control measure task-specific processes. In closing, I address the consequences of this conclusion. In particular, I emphasize the necessity (1) to stop generalizing the results beyond the specific task used in the study and (2) to stop using attentional control to explain group and individual differences.

show abstract

Section: Beyond the Methodological Challengesmentioning

confidence: 99%

Beyond Unity and Diversity: Tasks Used to Assess Attentional Control Measure Task-Specific Processes

Rey-Mermet

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…We anticipated that the Pavlovian bias in choice and reaction times would be modulated along with the change in uncontrollability, according to the model simulation experiments presented earlier. The virtual reality environment improves ecological validity [Parsons, 2015] and introduces gamification, which is known to also improve reliability of studies [Sailer et al, 2017, Kucina et al, 2023, Zorowitz et al, 2023, which is important in attempts to uncover potentially subtle biases. We used a hierarchical Bayesian estimation of model parameters, to increase the reliability across tasks (including the Go-No Go tasks [Zorowitz et al, 2023]).…”

Section: Experiments 3: Human Approach-withdrawal Conditioning Is Mod...mentioning

confidence: 99%

Balancing safety and efficiency in human decision making

Mahajan,

Tong,

Lee

et al. 2024

Preprint

View full text Add to dashboard Cite

The safety-efficiency dilemma describes the problem of maintaining safety during efficient exploration, and is a special case of the exploration-exploitation dilemma in the face of potentially catastrophic dangers. Conventional exploration-exploitation solutions collapse punishment and reward into a single signal, whereby early losses can be overcome by later gains. But the brain has a separate system for Pavlovian fear learning, suggesting a possible computational advantage to maintaining a specific fear memory during exploratory decision-making. In a series of simulations, we show here this promotes safe but efficient learning, and is optimised by arbitrating Pavlovian avoidance on instrumental decision-making according to uncertainty. In a human approach-withdrawal experiment, we show that this flexible avoidance model captures both choice and reaction times. These results show that the Pavlovian fear system has a more sophisticated role in decision-making that previously thought, by shaping flexible exploratory behaviour in a computationally precise manner.

show abstract

“…To derive a more accurate estimation of the average split-half reliability for each SPE measure, we synthesized these reliability coefficients via a meta-analytical approach. We weighed the reliability coefficients based on the trial numbers of each study since the number of trials typically significantly influences the reliability of cognitive experiments (Kucina et al, 2023) (see also Supplementary Fig. S7 for our exploratory analysis).…”

Section: Estimating the Reliability Of Spementioning

confidence: 99%

“…Third, we did not explicitly state in the preregistration report that we would perform a weighted average of the split-half reliabilities for all datasets. However, considering the significant impact of the number of trials on reliability (Kucina et al, 2023), during the formal analysis, we assigned different weights to each study based on the number of trials. Subsequently, we calculated a weighted average of the split-half reliabilities.…”

Section: Deviation From Preregistrationmentioning

confidence: 99%

“…In recent years, driven by a growing interest in employing cognitive tasks to assess individual differences, researchers have turned their attention to evaluating the reliability of cognitive tasks (e.g., Hedge et al, 2018;Kucina et al, 2023). However, existing findings have raised concerns about the reliability of many cognitive tasks (Karvelis et al, 2023;Rouder & Haaf, 2019), with a considerable body of research highlighting the moderate to low-level reliability found in the cognitive task measurements (Clark et al, 2022;Enkavi et al, 2019;Green et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Multiverse Assessment of the Reliability of the Self Matching Task as a Measurement of the Self-Prioritization Effect

Liu,

Hu,

Zheng

et al. 2023

Preprint

View full text Add to dashboard Cite

The Self Perceptual Matching Task (SPMT) is a widely used task to investigate the cognitive processes underlying the Self-Prioritization Effect (SPE), wherein performance is enhanced for self-associated stimuli compared to other-associated ones. Despite the wide use of SPMT, there is a lack of attention on its reliability assessment. This ignorance is concerning, given the prevalence of the reliability paradox in cognitive tasks: cognitive tasks demonstrate relatively low reliability when evaluating individual differences, though they produce robust experimen- tal effects. To fill this gap, this preregistered study investigated the reliability of SPMT using a multiverse approach, combining all possible indicators and baselines used to quantify SPE in SPMT. We examined the robustness and the reliability of 24 SPE measures across 17 datasets (N = 805). More specifically, we used a meta-analytical approach to estimate the robustness of SPE across datasets. We calculated the Split-Half Reliability (r) and Intraclass Correla- tion Coefficient (ICC2) for each SPE measure. Our findings revealed a robust experimental effect of SPE across datasets. However, when it came to individ- ual differences, SPE measures derived from Reaction Time (RT) and Efficiency exhibited relatively higher, compared to other SPE measures, but still unsatis- fied split-half reliability (approximately 0.6). Similarly, for the reliability across multiple time points, as assessed by ICC2, RT and Efficiency demonstrated low levels of test-retest reliability (close to 0.5). These findings uncovered the pres- ence of a reliability paradox in the context of SPMT-based SPE assessment. We discussed the implications of our findings for future studies.

show abstract

Calibration of cognitive tests to address the reliability paradox for decision-conflict tasks

Cited by 10 publications

References 59 publications

Beyond Unity and Diversity: Tasks Used to Assess Attentional Control Measure Task-Specific Processes

Beyond Unity and Diversity: Tasks Used to Assess Attentional Control Measure Task-Specific Processes

Balancing safety and efficiency in human decision making

A Multiverse Assessment of the Reliability of the Self Matching Task as a Measurement of the Self-Prioritization Effect

Contact Info

Product

Resources

About