Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

Abstract

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.

Confidence in Research Findings Depends on Theory

“…We agree with Almaatouq et al (2023) that the integration of disparate research findings is often inefficient or fails to occur entirely. However, their proposed solution is based on an inadequate but widely shared conception of the nature of theory and its importance for application.…”

Some problems with zooming out as science reform

“…At the same time, behavioral science is unlikely to change the world if we don't start taking heterogeneity of effects more seriously (Bryan et al, 2021). Integrative experiment design (Almaatouq et al, 2022) elevates heterogeneity by rendering explicit a larger design space from which experiments are sampled. By applying predictive modeling to test the generalization of surrogate models learned on portions of the space, it addresses the pervasive illusion that models chosen for their explanatory power also predict well (Yarkoni and Westfall, 2017).…”

“…Both multiverse analysis and integrative experiment design presuppose that our prior knowledge can take us just far enough to produce results more complex than current results sections, but not so complicated that we get overwhelmed. The "new kinds of theories" associated with integrative experimentation are meant to "capture the complexity of human behaviors while retaining the interpretability of simpler theories" (Almaatouq et al, 2022). There appears to be an implicit assumption that we can zoom out until we find the dimensionality that is considerably greater than the dimensionality of the problem implied by the status quo single experiment, but not so great as to be infinite, since then integrative experiment design could not resolve the current state of theoretical incommensurability.…”

“…

Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al, 2022) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments.

…”

Minority-group incubators and majority-group reservoirs support the diffusion of climate change adaptations

“…Specificity helps social scientists avoid sprawling verbal theories that may be mismatched to statistical models not suitable for causal inference [28][29][30]. Rigorous model development enables consistent experimental designs since experiments implemented with the same model are more likely to be commensurate; this would allow new climate change adaptation studies to multiply our understanding, instead of fracturing it across superficially disparate adaptation science sub-fields [31]. Agent-based, cultural evolutionary models are well-suited for developing mechanistic middle-range theories [32,33] that typically straddle theoretical paradigms where constituent cognitive and social processes take place across different dimensional scales, i.e., different spatial, temporal, and population scales [34].…”