Knowing how effective an intervention, treatment, or manipulation is and increasing replication rates: accuracy in parameter estimation as a partial solution to the replication crisis

Abstract

Objective: Although basing conclusions on confidence intervals for effect size estimates is preferred over relying on null hypothesis significance testing alone, confidence intervals in psychology are typically very wide. One reason may be a lack of easily applicable methods for planning studies to achieve sufficiently tight confidence intervals. This paper presents tables and freely accessible tools to facilitate planning studies for the desired accuracy in parameter estimation for a common effect size (Cohen's d). In addition, the importance of such accuracy is demonstrated using data from the Reproducibility Project: Psychology (RPP). Results: It is shown that the sampling distribution of Cohen's d is very wide unless sample sizes are considerably larger than what is common in psychology studies. This means that effect size estimates can vary substantially from sample to sample, even with perfect replications. The RPP replications' confidence intervals for Cohen's d have widths of around 1 standard deviation (95% confidence interval from 1.05 to 1.39). Therefore, point estimates obtained in replications are likely to vary substantially from the estimates from earlier studies. Conclusion:The implication is that researchers in psychology -and funders-will have to get used to conducting considerably larger studies if they are to build a strong evidence base.

Promoting ear plugs at music events: evaluation of the Celebrate Safe approach

“…However, the number of observations per event surpassed the pre-registration. The confidence intervals reported above show that the proportion of events visitors wearing ear plugs could be accurately estimated, but there is room for improvement regarding the predictor in the multilevel models (Peters and Crutzen 2020). The relative increase compared to the control condition and NNT are deemed impactful given the limited resources needed for the intervention activities.…”

We need to talk about reliability: making better use of test-retest studies for study design and interpretation

“…The reliability of outcome measures limits the range of standardised effect sizes which can be expected (although it can also increase their variance in small samples (Loken & Gelman, 2017), which is vital information for study design. Assuming constant underlying true scores, outcome measures with lower reliability have diminished power, meaning that more participants are required to reach the same conclusions, that the resulting parameter estimates are less precise (Peters & Crutzen, 2018), and that there is an increased risk of type M (magnitude) and type S (sign) errors (Gelman & Carlin, 2014). In extreme cases, if measured values are too dissimilar to the underlying ‘true’ values relative to any actual true differences between individuals, then a statistical test will have little to no possibility to infer meaningful outcomes: this has been analogised as “Gelman’s kangaroo” (Gelman, 2015; Wagenmakers & Gronau, 2017).…”

Meaningful change definitions: sample size planning for experimental intervention research

“…Once the MCD-based Cohen's d is established, the required sample size can be computed using power or AIPE computations described in a previous section (see also Cohen, 1988;Cumming, 2014;Faul et al, 2007;Peters and Crutzen, 2020). To illustrate the ball park, Table 2 shows the required sample sizes for control event rates of 5%, 25% and 50%, and MCDs of 2.5%, 5%, 10%, and 25%, the required sample sizes to obtain 80% and 95% power and to estimate the intervention effect with 95% confidence interval half-widths of d¼.10 and d¼.25.…”

“…First, if only one study has been conducted, given the small sample sizes and low statistical power that have long been customary in the psychological literature (Marszalek et al, 2011), the effect size estimate obtained in that study likely originates from an exceedingly wide sampling distribution. In other words, the observed effect size in that single study is to a large degree arbitrary, may differ considerably in the next direct replication (Peters & Crutzen, 2020), and as such does not provide a solid starting point for inferences about the likely population effect size (see also . If multiple studies have been conducted and their effect size estimates are extracted from the extant literature, meta-analysis may yield a more reliable estimate.…”