No abstract

Psychology endeavors to develop theories of human capacities and behaviors on the basis of a variety of methodologies and dependent measures. We argue that one of the most divisive factors in psychological science is whether researchers choose to use computational modeling of theories (over and above data) during the scientific-inference process. Modeling is undervalued yet holds promise for advancing psychological science. The inherent demands of computational modeling guide us toward better science by forcing us to conceptually analyze, specify, and formalize intuitions that otherwise remain unexamined—what we dub open theory. Constraining our inference process through modeling enables us to build explanatory and predictive theories. Here, we present scientific inference in psychology as a path function in which each step shapes the next. Computational modeling can constrain these steps, thus advancing scientific inference over and above the stewardship of experimental practice (e.g., preregistration). If psychology continues to eschew computational modeling, we predict more replicability crises and persistent failure at coherent theory building. This is because without formal modeling we lack open and transparent theorizing. We also explain how to formalize, specify, and implement a computational model, emphasizing that the advantages of modeling can be achieved by anyone with benefit to all.

SummaryData analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.

Psychology endeavors to develop theories of human capacities and behaviors based on a variety of methodologies and dependent measures. We argue that one of the most divisive factors in our field is whether researchers choose to employ computational modeling of theories (over and above data) during the scientific inference process. Modeling is undervalued, yet holds promise for advancing psychological science. The inherent demands of computational modeling guide us towards better science by forcing us to conceptually analyze, specify, and formalise intuitions which otherwise remain unexamined — what we dub “open theory”. Constraining our inference process through modeling enables us to build explanatory and predictive theories. Herein, we present scientific inference in psychology as a path function, where each step shapes the next. Computational modeling can constrain these steps, thus advancing scientific inference over and above stewardship of experimental practice (e.g., preregistration). If psychology continues to eschew computational modeling, we predict more replicability “crises” and persistent failure at coherent theory-building. This is because without formal modelling we lack open and transparent theorising. We also explain how to formalise, specify, and implement a computational model, emphasizing that the advantages of modeling can be achieved by anyone with benefit to all.