In two experiments, we explored the nature of the bias observed in the bat/ball problem of the cognitive reflection test (Frederick, 2005, J. Econ. Perspect., 19, 25), how to override it, and its relation to executive functioning. Based on the original bat/ball problem, we designed two additional isomorphic items. In Experiment 1, for four age groups, including 7‐, 11‐, and 15‐year‐olds and adults, we determined that the bias is related to the System 1 intervention; the performance in this item was not a matter of mathematical ability and it could be facilitated by changing the order in which the problems were presented. In Experiment 2, we determined that for 15‐year‐olds, good and bad performances in the item were related to executive functioning, particularly response‐distractor inhibition, updating information in working memory, and the regulation of attention; however, subtle differences were identified when the problem was performed in a facilitative context compared with a non‐facilitative context. The results indicated that cognitive abilities are a necessary but non‐sufficient condition to resolve the problem.