Sample size planning for the standardized mean difference: Accuracy in parameter estimation via narrow confidence intervals.

Abstract

Methods for planning sample size (SS) for the standardized mean difference so that a narrow confidence interval (CI) can be obtained via the accuracy in parameter estimation (AIPE) approach are developed. One method plans SS so that the expected width of the CI is sufficiently narrow. A modification adjusts the SS so that the obtained CI is no wider than desired with some specified degree of certainty (e.g., 99% certain the 95% CI will be no wider than ). The rationale of the AIPE approach to SS planning is given, as is a discussion of the analytic approach to CI formation for the population standardized mean difference. Tables with values of Keywords: sample size planning, standardized mean difference, accuracy in parameter estimation, power analysis, precision analysis One of the simplest measures of effect is the difference between two independent group means. It is this difference that is evaluated with the two-group t test to infer whether the population difference between two group means differs from some specified null value, which is generally set to zero. However, in the behavioral, educational, and social sciences, units of measurement are often arbitrary, different researchers might measure the same phenomenon with different scalings of the same instrument, or different instruments altogether might be used. Because of the lack of standard measurement scales and procedures for most behavioral, educational, and social phenomena, the ability to compare measures of effect across different situations has led many researchers to use standardized measures of effect. Measures of effect, or effect sizes, that are standardized yield scale-free numbers that are not wedded to a specific instrument or scaling metric. Given the measurement issues in behavioral, educational, and social research, such standardized effect sizes provide what is arguably the optimal way to estimate the size of an effect, along with its corresponding confidence interval, for a more communal knowledge base to be developed and so that the results from different studies can be compared more readily.A commonly used and many times intuitively appealing effect size is the standardized mean difference.

Confidence intervals and sample size calculations for the standardized mean difference effect size between two normal populations under heteroscedasticity

“…Although the two notions of expected width and assurance probability have been considered for sample size determination, the exact computations of E[W] and P {W ≤ ω} are more involved than those in Kelley and Rausch (2006) under a homogeneous variance framework. Naturally, the underlying exact distributional property of Welch's statistic V and estimated degrees of freedom b n should be incorporated into the sample size calculations as much as possible.…”

“…The other method provides the sample size needed to guarantee, with a given assurance probability, that the width of a confidence interval will not exceed the planned range. Essentially, the suggested sample size procedures are direct and heteroscedastic extensions of those considered in Kelley and Rausch (2006) under a homogeneous variance setting. This investigation updates and expands the current work in such a way that the findings not only improve the fundamental limitations of the existing method, but also reinforce the practice of measuring effect size in the context of heteroscedastic situations.…”

Methods for the Behavioral, Educational, and Social Sciences: An R package

“…level=.95, width=.30, degree.of. assurance=.99), which yields a necessary sample size of 362 (Kelley & Rausch, 2006). Thus, with 362 participants per group, a researcher can have 99% assurance that the width of the confidence interval for will be no larger than .30 units.…”

“…Example functions for sample size planning from the AIPE perspective that are available in MBESS are the standardized mean difference (Kelley & Rausch, 2006) using the ss.aipe.smd() function, the squared multiple correlation coefficient (fixed or random regressors) using the ss.aipe.R2() function (Kelley, 2007d), the coefficient of variation using the ss.aipe.cv() parameter to the scale of the effect size, using the confidence interval transformation principle that allows confidence limits from one metric to be transformed into confidence limits of another metric under certain conditions (e.g., Steiger, 2004;Steiger & Fouladi, 1997), confidence intervals for the population effect size of interest can be obtained. A set of functions exist that determine the limits of the confidence intervals directly (these functions call upon the noncentral functions discussed above) for several important and widely used effect size measures.…”

“…Although there are some functions within R and within certain R packages for planning sample size from a power analytic perspective, where the goal is to obtain results that reach statistical significance, the MBESS package contains functions for planning sample size from the power analytic perspective (e.g., Cohen, 1988;Kraemer & Thiemann, 1987;Lipsey, 1990;Murphy & Myors, 2004) as well as from the accuracy in parameter estimation (AIPE) perspective (e.g., Hahn & Meeker, 1991;Kelley, Maxwell, & Rausch, 2003;Kelley & Rausch, 2006;Kupper & Hafner, 1989). Whereas the goal of the power analytic approach is to plan sample size so that there is some desired probability of rejecting a false null hypothesis, the goal of the AIPE approach is to obtain a sufficiently narrow confidence interval with some desired probability.…”