Generalizability Crisis Meets Heterogeneity Revolution: Determining Under Which Boundary Conditions Findings Replicate and Generalize

Abstract

Intensive longitudinal studies typically examine phenomena that vary across time, individuals, contexts, and other boundary conditions. This poses challenges to the conceptualization and identification of replicability and generalizability, which refer to the invariance of research findings across samples and contexts as crucial criteria for trustworthiness. Some of these challenges are specific to intensive longitudinal studies, others are similarly relevant for the work with other complex datasets that contain multilayered sources of variation (individuals nested in different types of activities or organizations, regions, countries, etc.)This article opens with discussing the reasons why research findings may fail to replicate. We then analyze reasons why research findings may falsely appear to be non-replicable when in fact they were as such replicable, but lacked generalizability due to heterogeneity between samples, subgroups, individuals, time points, and contexts. Following that, we propose conceptual and methodological approaches to better disentangle non-replicability from non-generalizability and to better understand the exact causes of either problem. In particular, we apply Lakatos’s proposition to examine not only whether but under what boundary conditions a theory is a useful description of the world, to the question whether and under which conditions a research finding is replicable and generalizable. Not only will that contribute to a more systematic understanding of and research on replicability and generalizability in longitudinal studies and beyond, but it will also be a contribution to what has been called the heterogeneity revolution (Bryan et al., 2021; Moeller, 2021).

Social dynamics and affect: Investigating within-person associations in daily life using experience sampling and mobile sensing.

“…How do temporal dynamics in the regulation of social needs and social contact relate to momentary affect? In a combination of preregistered and exploratory analyses using a diverse set of assessment and analysis methods, we aimed to establish boundary conditions (Moeller et al, 2022) of social deprivation and social oversatiation in their effects on momentary affect. Across analyses, we consistently found that social oversatiation and a high desire to be alone were associated with a marked decline in affective well-being within persons over time.…”

Psychology Is a Property of Persons, Not Averages or Distributions: Confronting the Group-to-Person Generalizability Problem in Experimental Psychology

“…Overall, our research is consistent with recent critiques put forth in which some researchers (e.g., Richters, 2021;Speelman & McGann, 2020) have argued that there is a pervasive mismatch between psychological theorizing and the analytic procedures used for testing it-typical theorizing occurs at the person level, but analytic procedures operate at the group level. Over the past decade, much effort has gone toward correcting and promoting better statistical inferences (e.g., Lakens, 2021), but relatively fewer reform efforts have been aimed at appropriate psychological (i.e., scientific) inference (e.g., Liew et al, 2016;Moeller et al, 2022;Navarro, 2019) and development of explanatory formal theory (e.g., van Rooij & Baggio, 2021). The current research suggests that even if theorizing indeed improves, inference can still go wrong if familiar group-level statistical methods are privileged over person-level approaches.…”

Challenging the utility of polygenic scores for social science: Environmental confounding, downward causation, and unknown biology

“…That she says nothing about how underdetermination renders individuals qualitatively different from one another in terms of the functioning parameters and response dispositions of psychological structures and processes underlying their behavior? That she fails to mention that psychological heterogeneity is a pervasive, ubiquitous, defining characteristic of human functioning (Bryan, Tipton, & Yeager, 2021;McCaffrey, 2015;Moeller et al, 2022)? Or does Burt neglect to mention these things because they would call attention to something else she doesn't tell us: Namely, that the social science research she so vigorously champions but never gets around to describing is predicated on the logically implicit assumption that individuals are instead psychologically homogeneous, and that quantitative differences between them with respect to any particular pattern of overt functioning are produced by exactly the same psychological structures and processes operating in exactly the same ways in all individuals.…”