Psychology Is a Property of Persons, Not Averages or Distributions: Confronting the Group-to-Person Generalizability Problem in Experimental Psychology

Abstract

When experimental psychologists make a claim (e.g., “Participants judged X as morally worse than Y”), how many participants are represented? Such claims are often based exclusively on group-level analyses; here, psychologists often fail to report or perhaps even investigate how many participants judged X as morally worse than Y. More troubling, group-level analyses do not necessarily generalize to the person level: “the group-to-person generalizability problem.” We first argue for the necessity of designing experiments that allow investigation of whether claims represent most participants. Second, we report findings that in a survey of researchers (and laypeople), most interpret claims based on group-level effects as being intended to represent most participants in a study. Most believe this ought to be the case if a claim is used to support a general, person-level psychological theory. Third, building on prior approaches, we document claims in the experimental-psychology literature, derived from sets of typical group-level analyses, that describe only a (sometimes tiny) minority of participants. Fourth, we reason through an example from our own research to illustrate this group-to-person generalizability problem. In addition, we demonstrate how claims from sets of simulated group-level effects can emerge without a single participant’s responses matching these patterns. Fifth, we conduct four experiments that rule out several methodology-based noise explanations of the problem. Finally, we propose a set of simple and flexible options to help researchers confront the group-to-person generalizability problem in their own work.

Don’t SNARC me now! Intraindividual variability of cognitive phenomena – insights from the Ironman paradigm

“…In experimental psychology, researchers often draw conclusions from the robustness of effects at the group level to their robustness at the participant level (which is termed group-toperson generalizability problem by McManus et al, 2023); and they also assume that participant-level effects are stable over time and reflect the participant's typical behavior and thus a trait rather than a state characteristic. The goal of the current study was to provide a framework for evaluating these two types of generalizations.…”

“…Apart from this most obvious generalization, at least two more generalizations may be implicitly made by researchers at the same time. The first of these is the generalization that the group-level effect is reflected in each participant, which may be the case but is not necessarily warranted (see discussions on ecological fallacy in Fisher et al, 2018, and the group-to-person generalizability problem described by McManus et al, 2023). Until recently, this was even an explicit assumption in standard statistical methods, such as the general linear model.…”

How Not to Fool Ourselves About Heterogeneity of Treatment Effects

“…Lastly, researchers must ask if they are interested in heterogeneity between individuals or heterogeneity between groups. Many statistical methods, such as computing interaction terms between a treatment variable (e.g., receiving a stimulus payment) and a moderator variable (e.g., whether someone owns a home) estimate average effects at the group level, and do not reflect individual heterogeneity in response to treatment (McManus et al, 2023).…”