Effect Size and Power in Assessing Moderating Effects of Categorical Variables Using Multiple Regression: A 30-Year Review.

Abstract

The authors conducted a 30-year review (1969-1998) of the size of moderating effects of categorical variables as assessed using multiple regression. The median observed effect size (f(2)) is only .002, but 72% of the moderator tests reviewed had power of .80 or greater to detect a targeted effect conventionally defined as small. Results suggest the need to minimize the influence of artifacts that produce a downward bias in the observed effect size and put into question the use of conventional definitions of moderating effect sizes. As long as an effect has a meaningful impact, the authors advise researchers to conduct a power analysis and plan future research designs on the basis of smaller and more realistic targeted effect sizes.

The dimensions of written expression: Language group and gender differences

“…However, the group differences were marginally significant for organisation and not significant for vocabulary. These nonsignificant results might partly be due to the modest sample size, hence the low statistical power of the current study (see Aguinis, Beaty, Boik, & Pierce, 2005). Nonetheless, a number of methodological factors, including scoring procedures and task related factors, may also influence the scores and thereby, the strength of relations between the predictor measures and writing quality (see the section 4.4.).…”

The interaction between temperament and the family environment in adolescent substance use and externalizing behaviors: Support for diathesis–stress or differential susceptibility?

“…The main concern with moderation analysis is that it tends to lack power. The three most common problems leading to lack of power in moderation analysis include small sample size, with effect sizes for interactions that are often small (Aguinis, Beaty, Boik, & Pierce, 2005;Chaplin, 1991), low reliability of the predictor and/or moderator, which dramatically reduces the reliability of the interaction term, and restriction in range, where individuals in the studied population do not have the same probabilities of being selected for the sample (Aguinis, 1995; Aguinis & Stone- Romero, 1997;Aiken & West, 1991;McClelland & Judd, 1993).Once a significant interaction is found, the diathesis-stress and differential susceptibility models may be distinguished empirically by evaluating the pattern of the interaction. Statistical support for the diathesis-stress model comes from a pattern where an individual characteristic is related to an outcome and an ordinal (fan-shaped) interaction is found (Belsky et al, 2007).…”

The influence of entrepreneurs’ social referents on start-up size

“…This perspective has the advantage of facilitating multilevel analysis of cross-level interactions (Hundt and Sternberg 2014). In this regard, a multilevel random effects specification is more accurate than the multivariate methods (e.g., moderated multiple regressions) normally used in the management literature to estimate interaction effects (Aguinis et al 2005).…”