Some problems with zooming out as science reform

Abstract

Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al., 2022) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments. Attempts to debias data-driven inferences by “zooming out” are also challenged by the inseparability of the results sampled from a design space from the assumptions that produce them and the difficulty of rendering dependencies explicit without sacrificing interpretability.