How Big Are My Effects? Examining the Magnitude of Effect Sizes in Studies of Measurement Equivalence

Abstract

Recently, an effect size measure, known as d MACS, was developed for confirmatory factor analytic (CFA) studies of measurement equivalence. Although this index has several advantages over traditional methods of identifying nonequivalence, the scale and interpretation of this effect size are still unclear. As a result, the interpretation of the effect size is left to the subjective judgment of the researcher. To remedy this issue for other effect sizes, some have proposed guidelines for evaluating the magnitude of an effect based on the distribution of effect sizes in the literature. The goal of the current research was to develop similar guidelines for effect sizes of measurement nonequivalence and build on this work by also examining the practical importance of nonequivalence. Based on a review of past research, we conducted two simulation studies to generate distributions of effects sizes. Assuming the ideal scenario of invariant referent items, the results of these simulations were then used to develop empirical guidelines for interpreting nonequivalence and its effects on observed outcomes.

Age differences in Machiavellianism across the life span: Evidence from a large‐scale cross‐sectional study

“…However, as had been observed in prior research on these age groups (Lucas & Donnellan, 2011; Soto et al, 2008, 2011) d MACS detected violations of measurement equivalence for several items (adolescence vs. middle adulthood: Item (2) = .527, Item (4) = .668); adolescence versus. Late/late middle adulthood: Item (1) = .561, Item (2) = .783, Item (4) = .882; .40 = small effect; .60 = medium effect; .80 = large effect; Nye et al, 2018). Moreover, in both cases, the d MACS output contained a negative value in mean difference, meaning that the focal group (in this case adolescence) obtained a higher mean value due to differential item functioning between the compared age groups (Nye & Drasgow, 2011).…”

“…Prior to our main analyses, we examined whether the measures used in this study were invariant across age groups, genders, and income groups. In so doing, we employed change in comparative fix index (ΔCFI; Chen, 2007; Cheung & Rensvold, 2002; Hirschfeld & von Brachel, 2014) and d MACS ‐statistics (Nye, Bradburn, Olenick, Bialkow, & Drasgow, 2018; Nye & Drasgow, 2011) to assess measurement invariance (for a more detailed technical description see online supplement).…”

Exploring perceptions of meaningfulness in visual representations of bivariate relationships

“…Accordingly, in a context free setting, our study indicates that larger effect sizes than expected are required for participants to state that relationships among variables or differences between groups are meaningful. We say “than expected” since it has been common in the behavioural sciences to label r = .1/ d = .2 a small effect and r = .3/ d = .5 a moderate-sized effect (Cohen, 1988; Nye et al, 2018). As noted by Rosenthal & Rosnow (1984), a Pearson’s r equivalent to the Cohen’s d large effect ( d = 0.80) is r = 0.31 (p. 361).…”

Climbing the corporate ladder and within-person changes in narcissism: Reciprocal relationships over two decades

“…Each time after having placed additional constraints on the model, we tested change in model fit using ∆CFI. As Meade, Johnson, and Braddy (2008) argue, when ∆CFI values exceed .002, this suggests that at least one of the constrained parameters is non-invariant (see also Nye, Bradburn, Olenick, Bialko, & Drasgow, 2019). In case this happened, we explored potential causes of noninvariance using the modification indices.…”