Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test

Abstract

The cognitive reflection test (CRT) became popular for its impressive power to predict how well people reason and make decisions. Despite the popularity of the CRT, a major issue complicates its interpretation: The numerical nature of the CRT confounds reflection ability with mathematical ability. We have addressed this issue by developing the verbal CRT (CRT-V), a novel 10-item measure of cognitive reflection (https://osf.io/xehbv/), using nonmathematical problems with good statistical and psychometric properties and with low familiarity. First, we selected suitable items with relatively low familiarity and optimal difficulty as identified in two different populations (Studies 1 and 2) and with high content validity as judged by an expert panel (Study 3). Second, we demonstrated good criterion and construct validity for the test in different populations with a wide range of variables (Studies 4-6, 8) and a good internal consistency (Studies 4-8) and test-retest reliability (Study 7). The CRT-V was less associated with math anxiety, objective and subjective numeracy than the original CRT, and it was test equivalent across gender, age groups and administration setting. In contrast with the original CRT (Hedge's g = 0.29, 95% confidence interval [CI] [0.17, 0.40]), the CRT-V showed no gender differences (Hedge's g = −0.06, 95% CI [−0.18, 0.06]). The CRT-V can complement existing, numerical, tests of cognitive reflection.

The Effect of Mindfulness on Cognitive Reflection and Reasoning

“…Therefore, future research would benefit from employing different measures of cognitive reflection and reasoning. A more appropriate measure of cognitive reflection may be the 10-item verbal cognitive reflection test (CRT-V) which was recently developed by Sirota et al (2018). The CRT-V has been found to be a valid and reliable measure of cognitive reflection with the mean around the centre point of the summation index; therefore, 20 this test is less prone to floor effects than the CRT and the ECRT.…”

“…The CRT-V has been found to be a valid and reliable measure of cognitive reflection with the mean around the centre point of the summation index; therefore, 20 this test is less prone to floor effects than the CRT and the ECRT. Furthermore, as this test is less associated with numeracy, men and women have been found to perform equally well on this measure (Sirota et al, 2018). In order to assess reasoning ability, future studies may be improved by administering the Halpern Critical Thinking Assessment (HCTA) which scores participants on five dimensions of critical thinking, including verbal reasoning (Schuhfried, n.d.).…”

Analiza postrzegania prawdziwości informacji wśród studentów – podobieństwo super fake newsów do prawdziwych wiadomości

“…Przedstawiony podział i uzyskane korelaty z numeryczną (CRT; Toplak et al, 2014) i nie-numeryczną miarą analitycznego myślenia (CRT-V; Sirota et al, 2020) sugerują, iż skupienie się na analityczności myślenia nie wystarczy, by wyjaśnić, jak użytkownicy internetu postrzegają prawdziwość informacji. Czynniki psychologiczne, takie jak na przykład aktywnie otwarte myślenie (Stanovich & West, 1997), także odgrywają istotną rolę w wyjaśnianiu podatności na fake newsy.…”

Criterion Validity of Cognitive Reflection for Predicting Job Performance and Training Proficiency: A Meta-Analysis

“…Although the CRT (also called 3-item CRT) is the most famous test, other forms of the CRT were developed over the last decades. For instance, some researchers have developed larger CRTs (see, Sirota et al, 2020), and others have added new items to the CRT-3 items (e.g., Finucane and Gullion, 2010;Toplak et al, 2014;Primi et al, 2015). Recently, some researchers have shown an interest in the numerical content of the CRT, and they have developed verbal-CRTs (e.g., Thomson and Oppenheimer, 2016;Sirota et al, 2020).…”

“…For instance, some researchers have developed larger CRTs (see, Sirota et al, 2020), and others have added new items to the CRT-3 items (e.g., Finucane and Gullion, 2010;Toplak et al, 2014;Primi et al, 2015). Recently, some researchers have shown an interest in the numerical content of the CRT, and they have developed verbal-CRTs (e.g., Thomson and Oppenheimer, 2016;Sirota et al, 2020). These tests also trigger an immediate answer as in the 3-item CRT, and they might involve numbers in their statements, but mathematical calculations are not required in order to find the correct answer.…”