Harnessing the power of excess statistical significance: Weighted and iterative least squares.

Abstract

We introduce a new meta-analysis estimator, the weighted and iterated least squares (WILS), that greatly reduces publication selection bias (PSB) when selective reporting for statistical significance (SSS) is present. WILS is the simple weighted average that has smaller bias and rates of false positives than conventional meta-analysis estimators, the unrestricted weighted least squares (UWLS), and the weighted average of the adequately powered (WAAP) when there is SSS. As a simple weighted average, it is not vulnerable to violations in publication bias corrections models’ assumptions too often seen in application. WILS is based on the novel idea of allowing excess statistical significance (ESS), which is a necessary condition of SSS, to identify when and how to reduce PSB. We show in comparisons with large-scale preregistered replications and in evidence-based simulations that the remaining bias is small. The routine application of WILS in the place of random effects would do much to reduce conventional meta-analysis’s notable biases and high rates of false positives.

Beyond Random Effects: When Small-Study Findings Are More Heterogeneous

“…WILS uses UWLS to identify whether there is an excess of statistical significance in an area of research and discards those studies most responsible (Stanley & Doucouliagos, 2022; Stanley et al, 2021). Often, the remaining exaggeration is scientifically and practically insignificant (Stanley & Doucouliagos, 2022). When all studies in a meta-analysis are small, then any meta-analytic estimate should be interpreted with great caution (Ioannidis, 2005; Stanley et al, 2022).…”

“…If the systematic reviewer fears the effect of publication-selection bias and wishes to reduce it more aggressively, then there are versions of UWLS that also accomplish this goal—WAAP and weighted and iterative least squares (WILS). WILS uses UWLS to identify whether there is an excess of statistical significance in an area of research and discards those studies most responsible (Stanley & Doucouliagos, 2022; Stanley et al, 2021). Often, the remaining exaggeration is scientifically and practically insignificant (Stanley & Doucouliagos, 2022).…”

“…As discussed above, UWLS is a simple weighted average that allows heterogeneity to be correlated with standard errors. UWLS and FE have identical point estimates, but UWLS standard errors and CIs are larger when there is heterogeneity (Stanley & Doucouliagos, 2015, 2022). UWLS is easily calculated by a simple regression of the standardized effect size ( d i / S E i ) on its precision ( 1 / S E i ) when there is no intercept:…”

“…We conduct several simulations in which the key research dimensions (heterogeneity, the distribution of sample sizes, and mean effect sizes) are set to reflect the typical values found in large surveys of psychological research (Fraley & Vazire, 2014; Stanley et al, 2019). Supplement II.A in the Supplemental Material provides full details of simulations of VR-MRA using the core of past simulations’ code and design, previously posted on OSF (https://osf.io/eh974/; Stanley, 2019) and employed in studies of psychology (Stanley & Doucouliagos, 2022; Stanley et al, 2021) using other meta-analysis methods.…”

“…Our final simulation study closely followed the simulation design of VR-MRA, reported above and detailed in Section II of the Supplemental Material. The code for the core of the design was posted online in 2019 and used in other studies (Stanley, 2019; Stanley & Doucouliagos, 2022; Stanley et al, 2021). The most influential research dimensions are calibrated from large surveys of psychological research (Fraley & Vazire, 2014; Stanley et al, 2018)—for greater details, see Section II of the Supplemental Material.…”