Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection*

Abstract

This study investigates the small-sample performance of meta-regression methods for detecting and estimating genuine empirical effects in research literatures tainted by publication selection. Publication selection exists when editors, reviewers or researchers have a preference for statistically significant results. Meta-regression methods are found to be robust against publication selection. Even if a literature is dominated by large and unknown misspecification biases, precision-effect testing and joint precision-effect and meta-significance testing can provide viable strategies for detecting genuine empirical effects. Publication biases are greatly reduced by combining two biased estimates, the estimated meta-regression coefficient on precision (1/"Se") and the unadjusted-average effect. Copyright 2008 Blackwell Publishing Ltd and the Department of Economics, University of Oxford.

Publication selection in health policy research: The winner's curse hypothesis

“…Alternatively, WLS may be estimated from equation (1) ). Simulations show that the funnel-asymmetry and precision-effect tests can provide valid methods to identify publication selection and to test for the presence of a genuine effect beyond publication selection bias (Stanley, 2008). that affect reported results, and their omission can potentially bias these results.…”

“…(1) (Egger et al, 1997;Stanley, 2008). Where effect i is some estimated empirical effect, often an elasticity, and SE i is its standard error.…”

“…The conventional t-test of H 0 : α=0 from MRA model (1) is a valid, if low power, test for the presence of publication bias (Egger et al, 1997;Stanley, 2008). On the other hand, the t-test of H 0 : β=0 is a powerful test for the presence of an authentic empirical effect beyond publication selection (Stanley, 2005;Stanley, 2008).…”

“…On the other hand, the t-test of H 0 : β=0 is a powerful test for the presence of an authentic empirical effect beyond publication selection (Stanley, 2005;Stanley, 2008). These tests have been called the 'funnelasymmetry test' (FAT) and the 'precision-effect test' (PET), respectively, and can be used to confirm (or deny) objectively the subjective impression gleaned from a visual inspection of a funnel graph.…”

“…Recently developed meta-regression analysis (MRA) methods that are able to identify and correct publication selection bias are applied to exisiting findings and suggest confirmation of a 'winner's curse' in two areas of health research: health care income elasticities and prescription drug price elasticities (Gemmill et al, 2007;Stanley, 2008;Moreno et al, 2009;Costa-Font et al, 2011). That is, these methods allow investigation of potential publication bias in these two areas of health research and whether higher impact journals amplify this bias by reporting the largest effects.…”