Are There Reliable Qualitative Individual Difference in Cognition?

Abstract

In this paper we propose a new set of questions that focus on the direction of effects. In almost all studies the direction is important. For example, in a Stroop task it implies one theoretical explanation if incongruent words are slowed relative to congruent ones, but a completely different theoretical explanation if congruent words are slowed down relative to incongruent ones. We ask a 'does everybody' question, such as, 'does every individual show a Stroop effect?' Or, 'does every individual respond faster to loud as soft tones?' If all individuals truly have effects in the same direction that implicate a common theory we term the differences among them as quantitative individual differences. Conversely, if all individuals truly have effects in contrasting directions that implicate different theories we term the differences among them as qualitative individual differences. Here, we provide a users-guide to the question of whether individual differences are qualitative or quantitative.We discuss new software for assessment, and, more importantly, how the question impacts theory development in cognitive science. Our hope is that this mode of analysis is a productive tool in researchers' toolkits.

Who Is and Is Not "Average'"? Random Effects Selection with Spike-and-Slab Priors

“…The number of observations per person is particularly important because without this, the posterior inclusion probabilities tend to cluster around 0.5, giving no preference to whether an individual is "average" or not. The exact number of trials per person will depend on the ratio of between-person to residual variance, but it has been suggested that a minimum of 100 trials per person per condition may be needed (Rouder & Haaf, 2020). Researchers should bear this cost in mind when considering individual differences research.…”

“…The main advantages of our proposed methodology are that it allows the quantification of evidence for or against individual random effects, and because it can be fit using standard statistical software, it is flexible enough to be applied to a broad class of models (i.e., generalized linear mixed models). We seek to address a set of questions inspired by those in Grice et al (2020) and Rouder and Haaf (2020), but differ in an important way. With this framework we explicitly address the individual by providing a tool that is capable of answering which individuals are "average" and which ones are not?…”

“…In a similar spirit, Rouder and Haaf (2020) advocate for an approach with the aim of distinguishing situations where: all individuals have true effects in the same direction (quantitative differences), individuals have true effects in differing directions (qualitative differences), or all individuals have the same true effect (no differences) (also see Haaf & Rouder, 2017). They suggest fitting three mixed effects models, each of which corresponds to one of these circumstances, and comparing them with Bayes factors.…”

“…For instance, the person-centered effect sizes are general in that they can be applied across a wide variety of settings, but are computed in a somewhat adhoc manner with a focus on description. The approach in Rouder and Haaf (2020) allows for the quantification of evidence for or against qualitative individual differences, but ultimately relies on global descriptions of the quantitative, qualitative, and no individual differences models. Moreover, the proposed models are based on so-called g-priors.…”

The truth revisited: Bayesian analysis of individual differences in the truth effect

“…We find that researchers often stake out positions about individual differences a priori, and believe them with a surprising degree of confidence. One position we encounter is what we call the arbitrary diversity hypothesis (Rouder & Haaf, 2020). Accordingly, the human condition is so diverse that there must be people who deviate in all behaviors.…”