Statistical Power of Training Evaluation Designs

Abstract

Sample size requirements needed to achieve various levels of statistical power using posttest-only, gain-score, and analysis of covariance designs in evaluating training interventions have been developed. Results are presented which indicate that the power to detect true effects differs according to the type of design, the correlation between the pre-and posttest, and the size of the effect due to the training program. We show that the type of design and correlations between the pre-and posttest complexly determine the power curve. Finally, an estimate of typical sample sizes used in training evaluation design has been determined and reviewed to determine the power of the various designs to detect true effects, given this sample-size specification. Recommendations for type of design are provided based on sample size and projected correlations between pre-and posttest scores. Support for this paper was provided by the School of Management, University of Minnesota. Dr. Scott E. Maxwell provided statistical consultation regarding the statistical derivations found within the manuscript. COPYRIGHT 0 1985 PERSONNEL PSYCHOLOGY. INC 493

360‐degree feedback with systematic coaching: Empirical analysis suggests a winning combination

“…For instance, had we split the sample size in two to create a control group, we would have ruled out the ambiguity regarding potential alternative explanations, but in terms of power, "the true experimental design [would have been] strikingly inadequate" (Sackett & Mullen, 1993, p. 624). This is because the power of the experiment would have gone down from .98 as present in this study (for N = 80 [87 in our study] d = .5, r xy = .3) to .59 (for n e = 40 and n c = 40 [43.5 in our study], d = .5, r xy = .3), a 40% reduction in probability of correctly rejecting the null hypothesis when it is false (Arvey & Cole, 1989;Arvey et al, 1985;Cohen, 1988;Sackett & Mullen, 1993).…”

“…As in much previous research on training evaluations (e.g., Arvey, Cole, Hazucha, & Hartanto, 1985;Sackett & Mullen, 1993), the constraint of not including a control group in our study was the population of only N = 87 available in this organization. Sackett and Mullen (1993, p. 624) explicate this situation by noting that "… we see no ready mechanism for combating the low statistical power of the true experimental design in the setting where N is constrained.…”

Beyond Formal Experimental Design: Towards an Expanded View of the Training Evaluation Process

“…Recent writing by Arvey and coworkers (Arvey & Cole, 1989;Arvey, Cole, Hazucha, & Hartanto, 1985) has examined the statistical power of various evaluation designs. With small samples, power is often quite low.…”

“…Recent work (e.g., Arvey & Cole, 1989;Arvey, Cole, Hazucha, & Hartanto, 1985;) has made clear the sample size requirements needed to insure adequate statistical power. The findings are sobering: For many organizations, evaluation via formal experimental design is simply not feasible.…”

Comment l’image de la région d’origine influence-t-elle la qualité perçue de l’huile d’olive ?

“…Son niveau est influencé par plusieurs facteurs ; l'erreur α, la taille de l'échantillon, la taille de l'effet mesuré (Arvey et al, 1985 ;Baroudi et Orlikowski, 1989 ;Hallahan et Rosenthal, 1995). La puissance recommandée en sciences sociales est de l'ordre de 0.80 (Cohen, 1977 Dans notre expérimentation nous avons contrôlé le nom de la région d'origine (le même pour tous les produits testés, à l'intérieur de chaque pays), et nous avons varié l'information sur les dimensions de l'image de la région d'origine spécifique à l'huile d'olive.…”