There is no generalizability crisis

Lakens, Daniël; Tunç, Duygu Uygun; Mn, Tunç

doi:10.31234/osf.io/tm8jy

Cited by 17 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, while a methods section of an experimental paper might note that the participants were recruited from the subject pool at a particular university, it is not uncommon for research articles to report findings as if they apply to all of humanity (Henrich et al, 2010). According to this view, theories (and in fields such as experimental economics, formal models) are what help us understand the world, whereas experiments are merely instruments that enable researchers to test theories (Lakens et al, 2022;Levitt & List, 2007;Mook, 1983;Zelditch, 1969).…”

Section: The Universe Of Possible Experimentsmentioning

confidence: 99%

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

et al. 2022

View full text Add to dashboard Cite

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.

show abstract

Section: The Universe Of Possible Experimentsmentioning

confidence: 99%

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

et al. 2022

View full text Add to dashboard Cite

show abstract

“…For example, while a methods section of an experimental paper might note that the participants were recruited from the subject pool at a particular university, it is not uncommon for research articles to report findings as if they apply to all of humanity . According to this view, theories (and in fields such as experimental economics, formal models) are what help us understand the world, whereas experiments are merely instruments that enable researchers to test theories (Lakens, Uygun Tunç, & Necip Tunç, 2022;Levitt & List, 2007;Mook, 1983;Zelditch, 1969).…”

Section: The Universe Of Possible Experimentsmentioning

confidence: 99%

The elephant's other legs: What some sciences actually do

Baron

2024

Behav Brain Sci

View full text Add to dashboard Cite

Integrative experiments, as described, seem blindly empirical, as if the question of generality of effects could not be understood through controlled one-at-a-time experiments. But current research using such experiments, especially applied research, can resolve issues and make progress through understanding of cause–effect pathways, leaving to engineers the task of translating this understanding into practice.

show abstract

“…For example, while a methods section of an experimental paper might note that the participants were recruited from the subject pool at a particular university, it is not uncommon for research articles to report findings as if they apply to all of humanity . According to this view, theories (and in fields such as experimental economics, formal models) are what help us understand the world, whereas experiments are merely instruments that enable researchers to test theories (Lakens, Uygun Tunç, & Necip Tunç, 2022;Levitt & List, 2007;Mook, 1983;Zelditch, 1969).…”

Section: The Universe Of Possible Experimentsmentioning

confidence: 99%

Test many theories in many ways

Cyrus-Lai,

Tierney,

Uhlmann

2024

Behav Brain Sci

View full text Add to dashboard Cite

Demonstrating the limitations of the one-at-a-time approach, crowd initiatives reveal the surprisingly powerful role of analytic and design choices in shaping scientific results. At the same time, cross-cultural variability in effects is far below the levels initially expected. This highlights the value of “medium” science, leveraging diverse stimulus sets and extensive robustness checks to achieve integrative tests of competing theories.

show abstract

There is no generalizability crisis

Cited by 17 publications

References 4 publications

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

The elephant's other legs: What some sciences actually do

Test many theories in many ways

Contact Info

Product

Resources

About