Who detects and why: How individual differences in cognitive characteristics underpin different types of responses on reasoning tasks?

Abstract

When solving reasoning tasks such as the Cognitive reflection test (CRT) or the Belief bias syllogisms (BBS), people can do it in four different ways depending on whether they detected the conflict or not and whether they were accurate or not. Specifically, one can detect the conflict and be accurate, detect the conflict but be inaccurate, fail to detect the conflict but still be accurate or fail to detect the conflict and be inaccurate. In this study, we investigated whether the individual differences in intelligence, numerical ability, math knowledge and analytic thinking dispositions underpin these different approaches to solving reasoning tasks. Generally, we found that very accurate conflict non-detectors were the most intelligent, numerate and knowledgable of all the groups. On the contrary, inaccurate conflict non-detectors were the least intelligent, numerate and knowledgable, with the other two groups somewhere in between these two. However, the conclusions depended on what reasoning tasks were used and how the conflict detection was operationalized. We argue that these individual differences indicate that different people solve reasoning tasks in qualitatively different ways and that this calls in question the validity of reasoning tasks. Namely, we hypothesize that reasoning tasks could be measuring different constructs for different people. Specifically, we hypothesize and offer preliminary evidence that the CRT at least somewhat assesses reflection for those lower on cognitive abilities, but not at all for those very intelligent and numerate participants. We discuss how these findings relate to contemporary dual-process theories and validity of reasoning tasks and offer some speculations and suggestions.

Logical Intuition Is Not Really About Logic

“…Moreover, the results demonstrated that neither the so-called logical intuition nor pseudo-logical intuition are dependent on cognitive capacities. Contrary to some earlier findings (Erceg et al, 2019;Frey et al, 2017;Raoelison et al, 2020;Ricco et al, 2020;Šrol & De Neys, 2020;Thompson et al, 2018;Thompson & Johnson, 2014), we did not find evidence that logical intuitions were more prevalent amongst high-capacity reasoners; and novel to this experiment, we found the interference caused by pseudo conflict also did not vary as a function of cognitive ability.…”

“…The goal of Experiment 2 was to replicate the findings of Experiment 1 and to investigate the relationship between our pseudo-logical conflict effects and cognitive ability. Contrary to expectations, it has been observed that logical intuitions are most likely to be observed amongst high-rather than low-ability reasoners (Erceg et al, 2019;Frey et al, 2017;Hayes et al, 2020;Raoelison et al, 2020;Šrol & De Neys, 2020;Thompson et al, 2018;Thompson & Johnson, 2014; but see, Ghasemi et al, 2021;Howarth et al, 2018;Markovits et al, 2020;Nakamura & Kawaguchi, 2016). The explanation given for this relationship is that high ability reasoners have learned (at least simple) logical inferences to the point of automaticity (Stanovich, 2018;Thompson et al, 2018), so that they interfere with the ability to make belief-based judgement.…”

“…In Experiment 2, we used the same materials and design, but presented participants with two measures of cognitive capacity. Whilst several studies have showed that reasoners with higher cognitive capacities have increased intuitive sensitivity to logical validity (Erceg et al, 2019;Frey et al, 2017;Raoelison et al, 2020;Šrol & De Neys, 2020;Thompson et al, 2018;Thompson & Johnson, 2014), others have failed to find such a relationship between logical intuition capacity and cognitive ability (Ghasemi et al, 2021;Markovits et al, 2020;Morsanyi & Handley, 2012;Nakamura & Kawaguchi, 2016). The goal of this experiment was to test whether higher ability participants are similarly sensitive to both logic and pseudo-logic structures.…”

A diffusion model analysis of belief bias: Different cognitive mechanisms explain how cognitive abilities and thinking styles contribute to conflict resolution in reasoning

“…In particular, our results suggest that greater algorithmic abilities may facilitate the creation of decoupled problem representations (for similar findings, see Thompson et al, 2018;Thompson & Johnson, 2014), whereas a greater disposition to engage in critical thinking may facilitate the successful override of Type 1 responses. As such, our results are consistent with previous research that suggested that different individuals approach and solve reasoning tasks in qualitatively different ways (Erceg et al, 2019). All in all, results from individual-differences studies suggest that in different individuals, performance in reasoning tasks may be affected by different cognitive processes.…”

“…Moreover, Erceg et al (2020) also failed to find any association between CRT scores and reasoning performance in a belief bias task once individual differences in intelligence and numerical abilities were accounted for. In addition, recent findings have indicated that the CRT may only measure individual differences in cognitive reflection in less intelligent individuals who do not possess strong logical intuitions, but that it is not an accurate measure of cognitive reflection in more intelligent individuals with strong numerical abilities (Erceg et al, 2019). Hence, our findings chime in with other studies questioning the validity of the CRT as a measure of reflective thinking.…”