The InterModel Vigorish as a lens for understanding (and quantifying) the value of item response models for dichotomously coded items

Abstract

The deployment of statistical models, such as those used in item response theory (IRT), necessitates the use of indices that are informative about the degree to which a given model is appropriate for a specific data context. We introduce the InterModel Vigorish (IMV) as an index that can be used to quantify predictive accuracy for models of dichotomous item responses based on the improvement in predictive accuracy across two approaches to prediction (i.e., two models or a single model relative to prediction based on the mean). This index has a range of desirable features: it can be used for the comparison of non-nested models and its values are highly portable and generalizable. We use this fact to develop benchmarks for predictive performance based on a variety of simulated data contexts; these benchmarks can be used as priors in future work with the models we consider here. We also illustrate the utility of the IMV in empirical applications with a large variety of item response datasets.

The InterModel Vigorish for Model Comparison in Confirmatory Factor Analysis with Binary Outcomes

“…In contrast, the 6-factor model adeptly differentiated between item clusters and outperformed the other two models, with scale-level IMV values exceeding 0.03 and all item-level IMV values surpassing 0.04. These values align closely with the IMV values observed when contrasting the 2PL and 1PL models (Domingue et al, 2022), providing robust evidence in favor of the superior structure of the 6-factor model.…”

“…This value is similar to the benefits when replacing the 2PL with the 3PL model for the PISA math ability test (Domingue et al, 2021). Also, the IMVpp M0 , p M3 q is around 0.075, similar to the expected benefits (0.09) of a sports book for taking a wager (Domingue et al, 2022). By contrast, a ∆CFI of 0.02 might indicate the better fitting of an enhanced model, but the raw value remains hard to understand to what extent fit is increased.…”

“…The IMV allows for comparability across various contexts. In this example, the IMV values around 0.1-0.2 closely resemble the benefits of replacing the Classic Test Theory (CTT) model with a 1-parameter logistic (1-PL) item response model when predicting the data generated by the 1-PL model (Domingue et al, 2022). This similarity in the predictive value of these different approaches is perhaps not surprising given their known correspondence (McDonald, 2013).…”

“…One key advantage of the IMV is its portability across different contexts. For example, the IMV in comparing the CFA model with the prevalence model in the real-data analysis resembles the IMV in comparing the CTT model with the IRT model in Domingue et al (2022). In this sense, the IMV avoids the usage of specific cutoff values.…”

“…The IMV is a measure of the increase (or decrease) in predictive performance of a potentially 'enhanced model' in comparison to a baseline model when making predictions about data. The IMV index was first proposed to compare the prediction abilities of different models for binary outcomes, and has been further extended into item response models (Domingue et al, 2022). This index conveys an intuitive interpretation and is adaptable across diverse outcomes with varying prevalences.…”