Bulletproof Bias? Considering the Type of Data in Common Proportion of Variance Effect Sizes

Abstract

As effect sizes gain ground as important indicators of practical significance and as a meta-analytic tool, we must critically understand their limitations and biases. This project expands on research by @Okada2013, which highlighted the positive bias of eta squared and suggested the use of omega squared or epsilon for their lack of bias. These variance overlap measures were examined for potential bias in different data scenarios (i.e. truncated and Likert type data) to elucidate differences in bias from previous research. We found that data precision and truncation affected effect size bias, often lowering the bias in eta squared. This work expands our understanding of bias on variance overlap measures and allows researchers to make an informed choice about the type of effect to report given their research study. Implications for sample size planning and power are also discussed.