Generalizability in psychology research: Beware the grinch.

Abstract

Bauer (2023) lays out a strong argument that just as psychological scientists are having a reckoning over the replicability of our research, it is equally imperative that the field recognizes our long-standing problems with regard to generalizability. We agree with this assessment and highlight additional examples of research areas where these issues seem especially important with respect to application. We offer some thoughts about how the culture of research in psychology, including our collective tendency to minimize generalizability transparency, has maintained and compounded the problem. We make suggestions for a path forward that calls upon us to live up to the highest ideals of external validity.

Recognizing limits on the generalizability of findings of psychological science research.

“…Another point that was not taken up in the target article, but that is consistent with its general message, was noted by Mitchell and Shivde (2023), namely, that as a field, we seem to suffer from hesitation in discussing the limitations of our work broadly and potential limits on generalizability specifically. We seem to be hesitant, even though we know it is the right thing to do.…”

“…Across the commentaries, there is substantial convergence with the arguments in the target article (as well as in Yarkoni, 2022) that generalizability is a hallmark of psychological science and that at present, the field falls short of achieving this goal. The authors of the commentaries also highlight that these issues are especially critical when we have in mind not just behavior in the laboratory but in the world beyond it—that is, as we attempt to apply the findings of our research to classrooms, workplaces, and courtrooms, to name a few (Mitchell & Shivde, 2023; Otgaar et al, 2023). They reinforce the need for attention to cultural background, individual variability, methodological variations, and other sources of variance between and among participants that may limit generalizability of findings (Peterson, 2023; Prather, 2023).…”

“…Mitchell and Shivde (2023) also make the important point that issues of generalizability may vary at different levels of analysis or at different points in a research program. If the goal of a research project is to establish new ground by investigating a new phenomenon, then tests of whether the findings from the initial experiments generalize to other samples, populations, and methods may be premature.…”

“…That said, psychological science is sufficiently mature that it seems reasonable and appropriate to charge “admission” to the tent—if you want to stand under it, you must be authentic. If we do not seriously reflect on the authenticity of the work we do, then, as Mitchell and Shivde (2023) lament, we run the risk of going the way of alchemy.…”

“…Discussions of potential limitations on the generalizability of our findings are just as important as discussions of the strengths of our efforts; discussions of potential limits on generalizability may have as much if not more heuristic value for the field. As noted by Mitchell and Shivde (2023), we were taught to pay attention to and "own" the limitations of our work. We need to devote more effort to not just learning but to living that lesson.…”