The Thorny Relation Between Measurement Quality and Fit Index Cutoffs in Latent Variable Models

Abstract

Latent variable modeling is a popular and flexible statistical framework. Concomitant with fitting latent variable models is assessment of how well the theoretical model fits the observed data. Although firm cutoffs for these fit indexes are often cited, recent statistical proofs and simulations have shown that these fit indexes are highly susceptible to measurement quality. For instance, a root mean square error of approximation (RMSEA) value of 0.06 (conventionally thought to indicate good fit) can actually indicate poor fit with poor measurement quality (e.g., standardized factors loadings of around 0.40). Conversely, an RMSEA value of 0.20 (conventionally thought to indicate very poor fit) can indicate acceptable fit with very high measurement quality (standardized factor loadings around 0.90). Despite the wide-ranging effect on applications of latent variable models, the high level of technical detail involved with this phenomenon has curtailed the exposure of these important findings to empirical researchers who are employing these methods. This article briefly reviews these methodological studies in minimal technical detail and provides a demonstration to easily quantify the large influence measurement quality has on fit index values and how greatly the cutoffs would change if they were derived under an alternative level of measurement quality. Recommendations for best practice are also discussed.

Assessing the validity of a learning analytics expectation instrument: A multinational study

“…For the Spanish and Dutch translated versions of the SELAQ, the alternative fit indices do show the model fit to be acceptable. Although the RMSEA is high for the Spanish predicted expectation scale, the measurement quality is good and this is associated with the RMSEA functioning (McNeish et al, ). Thus, on the basis of these findings, it does support the use of the SELAQ to measure student expectations within these contexts.…”

“…As shown in the work of Xia (), it is inappropriate to generalize the Hu and Bentler criteria to occasions when the ULSMV estimator is used due to its dependency upon thresholds. In addition, the simulation study of McNeish, An, and Hancock () has shown these alternative fit indices (i.e., CFI and RMSEA) to be affected by the measurement quality of the model. Specifically, increased standardized factor loadings result in model fit indices that would be indicative of poor fit (Hancock & Mueller, ).…”

Development and validation of the School Climate Questionnaire for Secondary and High School Teachers (SCQ-SHST).

“…RMSEA value was just within the acceptable cutoff (.08) probably due to the large factor loadings size. Recent simulation studies have shown that in the presence of factor loadings above .70 the value of RMSEA tends to increase (McNeish, An, & Hancock, 2017;Savalei, 2012). Correlation between factors was moderate (r = .60).…”

Developmental cognitive neuroscience using Latent Change Score models: A tutorial and applications

“…SRMR or the RMSEA). Fit indices can be affected by a range of model and data properties including sample size, measurement quality, estimation method, misspecification and more (Fan, Thompson, & Wang, 1999;McNeish, An, & Hancock, 2017;Moshagen & Erdfelder, 2016). Competing models can be compared using traditional likelihood ratio test if models are nested (Neale, 2000), or specialized version of the LRT for non-nested models (Merkle, You, & Preacher, 2016).…”