Tell Us What You Really Think: A Think Aloud Protocol Analysis of the Verbal Cognitive Reflection Test

Abstract

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and non-mathematical verbal Cognitive Reflection Test (vCRT). Verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. The think-aloud protocols seemed to reflect business-as-usual performance: thinking aloud did not disrupt test performance compared to a control group. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of ‘reflection’ (as deliberate and conscious).

Zermelo Game

“…The so-called commognition paradigm by Sfard could help to interpret the use of the language of logic in classroom discussions and also students' thinking aloud. Byrd et al [41] study the effect of thinking aloud on reflection test performance: we are not sure that what we observed can be framed in thinking aloud protocols. Instead, our observations could be linked to particular theories of learning, such as the ICAP framework.…”

Analytic Atheism & Analytic Apostasy Across Cultures

“…Reflection test lure consideration and reflective responding. A substantial minority of people answer reflection tests correctly without actually reflecting (Byrd et al, 2023). To reduce such reflection test measurement error, we asked participants that didn't select the lure if they had considered the lure before selecting their answer.…”

“…And when studying causal relationships between reflection and religiosity, the so-called analytic atheist effect is not always observed (Yilmaz and Isler, 2019;Saribay, Yilmaz and Körpe, 2020). Worse, there are enough problems with standard protocols for measuring and manipulating reflection that we may need to reconsider whether many prior results actually support the conclusion that atheism is linked to analytic or reflective thinking (Byrd et al, 2023).…”

Avaliação em ambiente real de utilização de uma plataforma de criação de jornais escolares

“…No terceiro momento -cenário de utilização, foi pedida e incentivada a utilização do protocolo think aloud (Byrd et al, 2023), em que o participante é convidado a falar em voz alta sobre o que está a pensar durante a utilização da plataforma. A seguir, foi realizada uma exploração livre da plataforma por dois minutos, para facilitar a compreensão inicial da plataforma sem a preocupação de executar um conjunto de tarefas específicas.…”