Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science

Abstract

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

Effects of data preprocessing on results of the epidemiological analysis of coronary heart disease and behaviour-related risk factors

“…unexpectedly large variations in results across closely related studies, are only indirectly related to research, such as clinical factors and healthcare systems. Many reasons, however, originate from study designs, methodological choices, concept definitions, and observed data [ 1 , 2 ]. Reasons related to datasets include at least differences in sample size and representativeness of covariates.…”

Message Design Choices Don't Make Much Difference to Persuasiveness and Can't Be Counted On—Not Even When Moderating Conditions Are Specified

“…Attending to heterogeneity can be important for various purposes, such as serving as a goad to theoretical progress. For example, Linden and Hönekopp ( 2021 ) have argued that in psychological research, unexplained heterogeneity is a sign that the phenomenon is poorly understood, and hence reduction of such heterogeneity is a marker of research progress (see also Kenny and Judd, 2019 ). Where these purposes are central, an interest in moderator variables will lead to questions about whether moderators can explain variability.…”

Crowdsourced meta-analysis in psychological science: An open science approach to compiling and synthesizing scientific knowledge

“…A centralized and crowdsourced database of findings will not only increase the rate at which researchers absorb scientific information (i.e., consumptive efficiency; Thorngate, 1990), but will also scale alongside the exponentially growing literature to become more powerful as further findings are included (see Linden & Hönekopp, 2021 for a discussion of heterogeneity in meta-analysis). That is, the power and specificity of results will continue to improve as more data is added over time, and the range of competing hypothesis, control variables, and moderation effects that can be examined will likewise continue to increase.…”

“…As in any good meta-analysis, the result of integration is greater than the sum of its parts as effects that cannot be identified within primary studies are made visible when examined across studies. For example, the examination of heterogeneity (i.e., moderators such as context, culture, age, change over time) is a key role of meta-analysis (Linden & Hönekopp, 2021). While difficult to achieve in primary studies and even many smaller meta-analyses due to power constraints, meta-analyses based upon the full wealth of relevant research would suffer no such limitation.…”