The Reliability Paradox in Assessing Structural Relations Within Covariance Structure Models

Abstract

A two-step process is commonly used to evaluate data–model fit of latent variable path models, the first step addressing the measurement portion of the model and the second addressing the structural portion of the model. Unfortunately, even if the fit of the measurement portion of the model is perfect, the ability to assess the fit within the structural portion is affected by the quality of the factor–variable relations within the measurement model. The result is that models with poorer quality measurement appear to have better data–model fit, whereas models with better quality measurement appear to have worse data–model fit. The current article illustrates this phenomenon across different classes of fit indices, discusses related structural assessment problems resulting from issues of measurement quality, and endorses a supplemental modeling step evaluating the structural portion of the model in isolation from the measurement model.

The Role of Measurement Quality on Practical Guidelines for Assessing Measurement and Structural Invariance

“…As shown in Figure 6 (the dashed black line is a reference for the Cheung and Rensvold recommendation for DMNCI; the dashed gray line is a reference for DCFI), not until the factor loading magnitudes were about 0.55 was DCFI able to detect that constraining the structural parameters between groups was a misspecification when using criteria set forth by Cheung and Rensvold (2002). Similar to findings from Hancock and Mueller (2011) presented Figure 1, DCFI had a slight quadratic pattern for higher measurement quality conditions. As the factor loading magnitude increased, the indices became more and more sensitive to the misfit as seen by the upward trend for both values.…”

“…Most notably and most unexpectedly based on Hancock and Mueller (2011), DMNCI was found to be essentially unaffected by changes in measurement quality (and sample size) when testing measurement invariance, indicating that a single fit index criteria may be relatively stable across a wide range of conditions. Although there is debate within the methodological about the utility of using cutoff values to determine data-model fit (see, e.g., Barrett, 2007;Hayduk, Cummings, Boadu, Pazderka-Robinson, & Boulianne, 2007), should researchers subscribe to this philosophy (with all appropriate precautions), the recommended empirically derived cutoff values across conditions of measurement quality for DMNCI are 20.007 and 20.01 for the 5th and 1st percentile, respectively, based on the results of the simulation performed here.…”

“…The effect of factor reliability on GOFs has been previously documented in the context of single-group models by Hancock and Mueller (2011) and Saris, Satorra, and van der Veld (2009). In an illustrative study, Hancock and Mueller constructed a population covariance matrix based on a six-factor, longitudinal structural model with each factor having three manifest indicator variables that each had the same standardized factor loading value in the population.…”

“…Similar to the measurement invariance study above and to Hancock and Mueller (2011), the population values for the factor loadings were manipulated from 0.40 to 0.95 in increments of 0.05. Between-group misfit was induced through the specification of different population values for the structural parameters across groups, while the analysis model constrained the structural paths to be equivalent between groups.…”

Executive Functions: Formative Versus Reflective Measurement

“…It is noteworthy that the routine reporting of global fit for CFA models-which characterize how well model parameters reproduce the observed data-is not informative of maximal reliability. Good global model fit is not indicative of high-quality measurement; indeed, the converse is often true-excellent global fit can be obtained for CFA models that have very low maximal reliability (Hancock & Mueller, 2011). In practice, this implies that a model may do an excellent Downloaded by [Northeastern University] at 18:55 26 November 2014 job explaining the observed covariation among indicators (which may be trivially small) but that those indicators may represent a relatively poor approximation of the underlying latent construct.…”