Learning from Null Effects: A Bottom-Up Approach

Abstract

A critical barrier to generating cumulative knowledge in political science and related disciplines is the inability of researchers to observe the results from the full set of research designs that scholars have conceptualized, implemented, and analyzed. For a variety of reasons, studies that produce null findings are especially likely to be unobserved, creating biases in publicly accessible research. While several approaches have been suggested to overcome this problem, none have yet proven adequate. We call for the establishment of a new discipline-wide norm in which scholars post short “null results reports” online that summarize their research designs, findings, and interpretations. To address the inevitable incentive problems that earlier proposals for reform were unable to overcome, we argue that decentralized research communities can spur the broader disciplinary norm change that would bring advantage to scientific advance. To facilitate our contribution, we offer a template for these reports that incorporates evaluation of the possible explanations for the null findings, including statistical power, measurement strategy, implementation issues, spillover/contamination, and flaws in theoretical priors. We illustrate the template’s utility with two experimental studies focused on the naturalization of immigrants in the United States and attitudes toward Syrian refugees in Jordan.

Value Similarity and Norm Change: Null Effects and Backlash to Messaging on Same-Sex Rights in Uganda

“…The null effects of the African message and Western message on individuals’ perspectives towards LGB issues raise the question: are these findings driven by a genuine inability of the messages to induce change or are they instead the result of a threat to the internal validity of the study? To dissect the possible causes of this null finding, I turn to the framework for discussing null effects proposed by Alrababa’h et al (2022). This framework proposes that the following set of factors be considered as potential causes for a null finding: “(a) statistical power, (b) measurement strategy, (c) implementation issues, (d) spillover and contamination, and (e) flaws in the theoretical priors” (Alrababa’h et al, 2022, p. 5).…”

“…To dissect the possible causes of this null finding, I turn to the framework for discussing null effects proposed by Alrababa’h et al (2022). This framework proposes that the following set of factors be considered as potential causes for a null finding: “(a) statistical power, (b) measurement strategy, (c) implementation issues, (d) spillover and contamination, and (e) flaws in the theoretical priors” (Alrababa’h et al, 2022, p. 5).…”

“…This paper proceeds by contextualizing the status of LGB rights and attitudes in Sub-Saharan Africa; identifying theoretical priors of norm change; describing the survey experiment used to assess the effect of informational signals; and reporting results of the experiment. I then discuss the null effects of the messages on positions towards LGB issues, utilizing the framework proposed by Alrababa'h et al (2022) to diagnose potential explanations for these results. I show that messages about LGB issues from African sources conveyed new information about the status of LGB rights in the region but that messages from Western sources did not contain new information due to a high degree of baseline knowledge.…”

Can trade and security alliance help reduce interstate war?

“…This null finding should be noted as an effort to inseminate scientific truth, as Laitin [28 page 5] points out that "we need to incentivize the publication of replications and null results." Recently, Alrababa'h et al [29] also underscore the importance of null findings in pursuing scientific discovery.…”

“…This null finding is also verified by robustness tests such as a Heckman selection model and a generalized method of moments model to control for endogeneity bias. This null finding should be noted as an effort to inseminate scientific truth, as Laitin [ 28 page 5] points out that “we need to incentivize the publication of replications and null results.” Recently, Alrababa’h et al [ 29 ] also underscore the importance of null findings in pursuing scientific discovery.…”