Dynamic Fit Index Cutoffs for Hierarchical and Second-Order Factor Models

Abstract

A recent review found that 11% of published factor models are hierarchical with second-order factors. However, dedicated recommendations for evaluating hierarchical model fit has yet to emerge. Traditional benchmarks like RMSEA<0.06 or CFI>0.95 are often consulted, but they were never intended to generalize to hierarchical models. Through simulation, we show that traditional benchmarks perform poorly at identifying misspecification in hierarchical models. This corroborates previous studies showing that traditional benchmarks do not maintain optimal sensitivity to misspecification as model characteristics deviate from those used to derive the benchmarks. Instead, we propose a hierarchical extension to the dynamic fit index (DFI) framework, which automates custom simulations to derive cutoffs with optimal sensitivity for specific model characteristics. In simulations to evaluate performance, results showed that the hierarchical DFI extension routinely exceeded 95% classification accuracy and 90% sensitivity to misspecification whereas traditional benchmarks rarely exceeded 50% classification accuracy and 20% sensitivity. though they were never intended to be generalized in this way. For instance, Hu and Bentler (1998) state, "our findings suggest that the performance of fit indices is complex and that additional research with a wider class of models and conditions is needed" (p. 446) and continue by writing, "Further work should be performed to explore the limits of generalizability in various ways, for example, across types of structural models and overparameterized models" (p. 450).The main goal of this paper is to provide researchers with a more refined set of fit index cutoffs for hierarchical factor models within the dynamic fit index (DFI) framework . DFI is an automated version of ideas proposed Millsap (2007Millsap ( , 2013 to perform custom simulations that derive fit index cutoffs tailored to the specific model being evaluated.Such customization limits concerns about generalizing cutoffs derived from other conditions and maintains optimal classification properties of cutoffs regardless of the model's characteristics.The DFI framework currently encompasses one-factor and multiple-factor confirmatory factor analysis (CFA) models, but it has not been extended to hierarchical factor models despite their widespread use in fields such as personality (e.g.