Predictive Fit Metrics for Item Response Models

Abstract

The fit of an item response model is typically conceptualized as whether a given model could have generated the data. In this study, for an alternative view of fit, “predictive fit,” based on the model’s ability to predict new data is advocated. The authors define two prediction tasks: “missing responses prediction”—where the goal is to predict an in-sample person’s response to an in-sample item—and “missing persons prediction”—where the goal is to predict an out-of-sample person’s string of responses. Based on these prediction tasks, two predictive fit metrics are derived for item response models that assess how well an estimated item response model fits the data-generating model. These metrics are based on long-run out-of-sample predictive performance (i.e., if the data-generating model produced infinite amounts of data, what is the quality of a “model’s predictions on average?”). Simulation studies are conducted to identify the prediction-maximizing model across a variety of conditions. For example, defining prediction in terms of missing responses, greater average person ability, and greater item discrimination are all associated with the 3PL model producing relatively worse predictions, and thus lead to greater minimum sample sizes for the 3PL model. In each simulation, the prediction-maximizing model to the model selected by Akaike’s information criterion, Bayesian information criterion (BIC), and likelihood ratio tests is compared. It is found that performance of these methods depends on the prediction task of interest. In general, likelihood ratio tests often select overly flexible models, while BIC selects overly parsimonious models. The authors use Programme for International Student Assessment data to demonstrate how to use cross-validation to directly estimate the predictive fit metrics in practice. The implications for item response model selection in operational settings are discussed.

The InterModel Vigorish as a lens for understanding (and quantifying) the value of item response models for dichotomously coded items

“…Of particular interest is the observation that IMV(2PL,3PL) tends to be near zero as the 2PL can effectively approximate guessing via adjustment to difficulty and discrimination parameters in a way that makes the resulting out-of-sample 2PL estimates of x p ij highly comparable to those produced by the 3PL. Results here are similar to others [23] in suggesting that the 3PL might have limited utility in many settings. We also use the inherently comparative nature of the IMV to introduce the Oracle and the Overfit values.…”

“…Frequently, analysis of out-of-sample data for the purposes of model selection has focused on which approaches allow one to identify the data generating model. In contrast, they [23] argue that we should instead be asking which models are maximally predictive of out-of-sample data and favoring those, irrespective of whether they are also the data generating model (although we would clearly expect them to be in some cases). In the sense that it is designed to gauge the quality of prediction in out-of-sample tests, the IMV is a "predictive index".…”

“…An early example of this kind of analysis in psychometric settings emphasized a version of the out-of-sample log-likelihood as performing better than many alternatives [10]. 1 A more recent paper [23] introduced one insight that is key for our purposes. Frequently, analysis of out-of-sample data for the purposes of model selection has focused on which approaches allow one to identify the data generating model.…”

“…We consider a random sample of 10,000 respondents in datasets with large numbers of respondents. To compute the IMV, we implement the "missing response" paradigm [23] by splitting the data into N " 4 folds. For a given fold, we use all responses not in that fold to produce estimates of ability and item parameters and then combine those to generate predictions for the responses in the fold.…”

A Comparative Study of Item Response Theory Models for Mixed Discrete-Continuous Responses

“…Parameter Estimation . Parameter estimation was conducted using an out-of-sample prediction approach through cross-validation ( Stenhaug and Domingue 2022 ). The complete dataset was divided randomly into six folds, ensuring each fold included at least one response from every participant with at least six responses (98.9% of respondents).…”