Why hypothesis testers should spend less time testing hypotheses

Abstract

For almost half a century, Paul Meehl educated psychologists about how the mindless use of null-hypothesis significance tests made research on theories in the social sciences basically uninterpretable (Meehl, 1990). In response to the replication crisis, reforms in psychology have focused on formalising procedures for testing hypotheses. These reforms were necessary and impactful. However, as an unexpected consequence, psychologists have begun to realise that they may not be ready to test hypotheses. Forcing researchers to prematurely test hypotheses before they have established a sound ‘derivation chain’ between test and theory is counterproductive. Instead, various non-confirmatory research activities should be used to obtain the inputs necessary to make hypothesis tests informative. Before testing hypotheses, researchers should spend more time forming concepts, developing valid measures, establishing the causal relationships between concepts and their functional form, and identifying boundary conditions and auxiliary assumptions. Providing these inputs should be recognised and incentivised as a crucial goal in and of itself. In this article, we discuss how shifting the focus to non-confirmatory research can tie together many loose ends of psychology’s reform movement and help us lay the foundation to develop strong, testable theories, as Paul Meehl urged us to.

Disability and the COVID-19 Pandemic: A Survey of Individuals With Traumatic Brain Injury

“…We expect that these data will provide a baseline as to how individuals with TBI are responding to the pandemic and will begin a derivation chain informing future hypothesis-driven research in this area. 28…”

A (More) Behavioural Science of Personality in the Age of Multi–Modal Sensing, Big Data, Machine Learning, and Artificial Intelligence

“…For them, machine learning is simply useful because it can already provide solutions and show what ‘works’ (and what not)—and that irrespective of hypothesis testing or building towards theory. Nonetheless, both types of scientists would (hopefully) agree that we need proper conceptual precision, logical reasoning, valid measures, and good data quality to be at all able to properly ‘test’ any hypotheses (see Scheel, Tiokhin, Isager, & Lakens, 2020) or solve any problem, respectively. There are recent calls for more formalization and theory–building in psychology (e.g.…”

A decade of theory as reflected in Psychological Science (2009-2019)

“…If the majority of research in a top journal is not explicitly testing predictions derived from theory, then perhaps it exists to explore and describe interesting effects. There is certainly nothing wrong with a descriptive approach, and this aim of psychology has been suggested for at least half a century (20,42,43).…”