Dealing With Omitted and Not-Reached Items in Competence Tests

Abstract

Data from competence tests usually show a number of missing responses on test items due to both omitted and not-reached items. Different approaches for dealing with missing responses exist, and there are no clear guidelines on which of those to use. While classical approaches rely on an ignorable missing data mechanism, the most recently developed model-based approaches account for nonignorable missing responses. Model-based approaches include the missing propensity in the measurement model. Although these models are very promising, the assumptions made in these models have not yet been tested for plausibility in empirical data. Furthermore, studies investigating the performance of different approaches have only focused on one kind of missing response at once. In this study, we investigated the performance of classical and model-based approaches in empirical data, accounting for different kinds of missing responses simultaneously. We confirmed the existence of a unidimensional tendency to omit items. Indicating nonignorability of the missing mechanism, missing tendency due to both omitted and not-reached items correlated with ability. However, results on parameter estimation showed that ignoring missing responses was sufficient to account for missing responses, and that the missing propensity was not needed in the model. The results from the empirical study were corroborated in a complete case simulation.

Commentary: On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

“…First, T&B's result seems somewhat surprising since the finding on the performance of incorrect scoring stands in stark contrast to other published research on this approach (Lord, 1974; De Ayala et al, 2001; Rose et al, 2010; Pohl et al, 2014) which show that incorrect scoring results in highly biased parameter estimates whenever missing values do not only occur on otherwise incorrect responses. Second, note that scoring missing values as incorrect results in a different definition of the target ability for different subgroups.…”

Dealing With Item Nonresponse in Large‐Scale Cognitive Assessments: The Impact of Missing Data Methods on Estimated Explanatory Relationships

“…Since some of the reasons for missing values include inability to answer the item correctly as well as other personality states and traits (cf. Jakwerth et al, 1999;Köhler et al, 2015a;Koretz et al, 1993;Pohl et al, 2014;Rose et al, 2010), we generated the missing data such that there would be dependence on ability and an explanatory variable in order to examine whether disregard of these relationships affects the parameter estimates of the structural model.…”

“…Current practices in low-stakes educational large-scale achievement tests involve treating unplanned missing values as incorrect or fractionally correct responses or ignoring them in the scaling (see, e.g., PISA, Adams & Wu, 2002;TIMSS [Third International Mathematics and Science Study], Martin, Gregory, & Stemler, 2000; NAEP [National Assessment of Educational Progress], Allen, Donoghue, & Schoeps, 2001;NEPS [National Educational Panel Study], Pohl & Carstensen, 2012). Research on these types of missing data approaches showed bias on item and person parameter estimates when missing values were scored as incorrect (Culbertson, 2011;De Ayala, Plake, & Impara, 2001;Finch, 2008;Hohensinn & Kubinger, 2011;Holman & Glas, 2005;Pohl, Gräfe, & Rose, 2014;Rose et al, 2010). The method of fractionally correct scoring performed slightly better but also resulted in bias, especially when missing values were MNAR (De Ayala et al, 2001;Finch, 2008).…”

“…The joint distribution between θ and ξ is considered in the measurement model, thus taking nonignorable missing values into account (Holman & Glas, 2005;O'Muircheartaigh & Moustaki, 1999). Several studies have demonstrated that model-based approaches perform well in retrieving adequate item and person parameter estimates, even when missing values are MNAR (Holman & Glas, 2005;O'Muircheartaigh & Moustaki, 1999;Pohl et al, 2014;Rose, 2013;Rose et al, 2010).…”

Conditioning factors of test-taking engagement in PIAAC: an exploratory IRT modelling approach considering person and item characteristics

“…In fact, omitted responses are often at least partially due to a lack of test-taking motivation (Jakewerth et al 1999;Wise and DeMars 2005). Latent variable modelling of omission propensity has revealed a negative relation with ability, that is, stronger test-takers omit fewer items (Holman and Glas 2005;Pohl et al 2013). Köhler et al (2015) provided some evidence that people without a migration background and with higher levels of education exhibit a lower omission propensity.…”