Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration

Abstract

Social scientists generally enjoy substantial latitude in selecting measures and models for hypothesis testing. Coupled with publication and related biases, this latitude raises the concern that researchers may intentionally or unintentionally select models that yield positive findings, leading to an unreliable body of published research. To combat this "fishing" problem in medical studies, leading journals now require preregistration of designs that emphasize the prior identification of dependent and independent variables. However, we demonstrate here that even with this level of advanced specification, the scope forfishing is considerable when there is latitude over selection of covariates, subgroups, and other elements of an analysis plan. These concerns could be addressed through the use of a form of comprehensive registration. We experiment with such an approach in the context of an ongoing field experiment for which we drafted a complete “mock report" of findings using fake data on treatment assignment. We describe the advantages and disadvantages of this form of registration and propose that a comprehensive but non-binding approach be adopted as a first step to combat fishing by social scientists. Likely effects of comprehensive but non-binding registration are discussed, the principle advantage being communication rather than commitment, in particular that it generates a clear distinction between exploratory analyses and genuine tests.

Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments

“…The covariate-by-covariate search for backlash is fraught with the risk of making false discoveries (Gelman and Loken 2016; Humphreys et al 2013; Kerr 1998). In order to guard against these pitfalls, we conduct a holistic analysis of treatment effect heterogeneity using a method that considers all measured moderators in a single model.…”

The Neighborhood Impacts of Local Infrastructure Investment: Evidence from Urban Mexico

“…Our marginal effect of −0.152 on the number of assaults in the past year would indicate that the 118,000 households in treatment polygons suffered a total of almost 18,000 fewer physical assaults as a result of the program. While highly significant, these outcomes need to be corrected for multiple inference because they have been selected from the much broader potential set of social capital variables (Casey, Glennerster, and Miguel 2012;Humphreys, Sanchez de la Sierra, and van der Windt 2013). At the bottom of online Appendix Table A5, we therefore illustrate several different ways of penalizing ourselves for this fishing using Anderson's False Discovery Rate sharpened q-values (Anderson 2008).…”

Just Say No to p < x (∀x ∊ (0, 1]), *s and Other Evil Things

“…A conclusion from this might be that even observational work would need to be preregistered (see Burlig ; Humphreys et al. ). Gelman and Loken (: 460), in their more broad‐sweeping discussion, mention preregistration as one way out of the conundrum, highlight, however, also a more profound problem, namely “the garden of forking paths.” Each research from its initiation all the way to its publication is based on choices at numerous forking paths.…”