A Mixture IRTree Model for Performance Decline and Nonignorable Missing Data

Huang, Hung‐Yu

doi:10.1177/0013164420914711

Cited by 14 publications

(7 citation statements)

References 62 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Latent ignorability [ 19 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] is one of the weakest nonignorable missingness mechanisms. Latent ignorability weakens the assumption of ignorability for MAR data.…”

Section: Statistical Models For Handling Missing Item Responsesmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

Missing item responses are prevalent in educational large-scale assessment studies such as the programme for international student assessment (PISA). The current operational practice scores missing item responses as wrong, but several psychometricians have advocated for a model-based treatment based on latent ignorability assumption. In this approach, item responses and response indicators are jointly modeled conditional on a latent ability and a latent response propensity variable. Alternatively, imputation-based approaches can be used. The latent ignorability assumption is weakened in the Mislevy-Wu model that characterizes a nonignorable missingness mechanism and allows the missingness of an item to depend on the item itself. The scoring of missing item responses as wrong and the latent ignorable model are submodels of the Mislevy-Wu model. In an illustrative simulation study, it is shown that the Mislevy-Wu model provides unbiased model parameters. Moreover, the simulation replicates the finding from various simulation studies from the literature that scoring missing item responses as wrong provides biased estimates if the latent ignorability assumption holds in the data-generating model. However, if missing item responses are generated such that they can only be generated from incorrect item responses, applying an item response model that relies on latent ignorability results in biased estimates. The Mislevy-Wu model guarantees unbiased parameter estimates if the more general Mislevy-Wu model holds in the data-generating model. In addition, this article uses the PISA 2018 mathematics dataset as a case study to investigate the consequences of different missing data treatments on country means and country standard deviations. Obtained country means and country standard deviations can substantially differ for the different scaling models. In contrast to previous statements in the literature, the scoring of missing item responses as incorrect provided a better model fit than a latent ignorable model for most countries. Furthermore, the dependence of the missingness of an item from the item itself after conditioning on the latent response propensity was much more pronounced for constructed-response items than for multiple-choice items. As a consequence, scaling models that presuppose latent ignorability should be refused from two perspectives. First, the Mislevy-Wu model is preferred over the latent ignorable model for reasons of model fit. Second, in the discussion section, we argue that model fit should only play a minor role in choosing psychometric models in large-scale assessment studies because validity aspects are most relevant. Missing data treatments that countries can simply manipulate (and, hence, their students) result in unfair country comparisons.

show abstract

Section: Statistical Models For Handling Missing Item Responsesmentioning

confidence: 99%

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Robitzsch

2021

EJIHPE

View full text Add to dashboard Cite

show abstract

“…In contrast, higher interest in continuous changes of response strategies within a measurement situation exists in item response modeling outside the RS literature. In the research field of performance decline, which describes a decreasing probability of correct responses for achievement items at the end of a test (for an overview, see List et al, 2017), the gradual process change model by Wollack and Cohen (2004) and Goegebeur et al (2008) is a prominent model for generating and analyzing smooth changes in response strategies (e.g., Huang, 2020;Jin & Wang, 2014;Shao et al, 2016;Suh et al, 2012). In their approach, the response process of random guessing gradually takes over from trait-based problem-solving, and linear as well as curvilinear trajectories can be captured.…”

Section: Modeling Heterogeneity Of Response Processesmentioning

confidence: 99%

Dynamic Response Strategies: Accounting for Response Process Heterogeneity in IRTree Decision Nodes

Merhof

Meiser

2023

Psychometrika

View full text Add to dashboard Cite

It is essential to control self-reported trait measurements for response style effects to ensure a valid interpretation of estimates. Traditional psychometric models facilitating such control consider item responses as the result of two kinds of response processes—based on the substantive trait, or based on response styles—and they assume that both of these processes have a constant influence across the items of a questionnaire. However, this homogeneity over items is not always given, for instance, if the respondents’ motivation declines throughout the questionnaire so that heuristic responding driven by response styles may gradually take over from cognitively effortful trait-based responding. The present study proposes two dynamic IRTree models, which account for systematic continuous changes and additional random fluctuations of response strategies, by defining item position-dependent trait and response style effects. Simulation analyses demonstrate that the proposed models accurately capture dynamic trajectories of response processes, as well as reliably detect the absence of dynamics, that is, identify constant response strategies. The continuous version of the dynamic model formalizes the underlying response strategies in a parsimonious way and is highly suitable as a cognitive model for investigating response strategy changes over items. The extended model with random fluctuations of strategies can adapt more closely to the item-specific effects of different response processes and thus is a well-fitting model with high flexibility. By using an empirical data set, the benefits of the proposed dynamic approaches over traditional IRTree models are illustrated under realistic conditions.

show abstract

“…Consequently, a larger π j suggests that test‐takers are more likely to randomly endorse an option when responding to item j . Similar to the IRTree models for MC items (e.g., Debeer et al, 2017; Huang, 2020), the MixNRM can also be displayed using a tree‐based diagram. Figure 1 illustrates the hierarchical structure of the proposed MixNRM‐RG for test‐taker i answering item j with four options.…”

Section: The Proposed Modelmentioning

confidence: 99%

“…The current study assumes that the latent membership x ij follows a Bernoulli distribution with parameter π j , which represents the marginal probability of random guessing on item j. Consequently, a larger π j suggests that test-takers are more likely to randomly endorse an option when responding to item j. Similar to the IRTree models for MC items (e.g., Debeer et al, 2017;Huang, 2020), the MixNRM can also be displayed using a tree-based diagram. Figure 1 illustrates the hierarchical structure of the proposed MixNRM-RG for test-taker i answering item j with four options.…”

Section: The Proposed Modelmentioning

confidence: 99%

Exploring the Impact of Random Guessing in Distractor Analysis

Jin¹,

Siu²,

Huang

2022

J Educational Measurement

View full text Add to dashboard Cite

Multiple-choice (MC) items are widely used in educational tests. Distractor analysis, an important procedure for checking the utility of response options within an MC item, can be readily implemented in the framework of item response theory (IRT). Although random guessing is a popular behavior of test-takers when answering MC items, none of the existing IRT models for distractor analysis have considered the influence of random guessing in this process. In this article, we propose a new IRT model to distinguish the influence of random guessing from response option functioning. A brief simulation study was conducted to examine the parameter recovery of the proposed model. To demonstrate its effectiveness, the new model was applied to the mathematics tests of the Hong Kong Diploma of Secondary Education Examination (HKDSE) from 2015 to 2019. The results of empirical analyses suggest that the complexity of item contents is a key factor in inducing students' random guessing. The implications and applications of the new model to other testing situations are also discussed.Multiple-choice (MC) questions are undoubtedly the most popular format used in existing academic assessments. An MC item is composed of a stem and several (usually 2−5) response options (Lord, 1977), from which test-takers are expected to select the only correct option or the best option among the given ones. The incorrect options are also referred to as "distractors." The design of distractors is directly related to the difficulty of an MC item. A good MC item should have an accurate identification of the correct options and distractors (Fellenz, 2004). Moreover, distractors should be sufficiently attractive. A distractor that fails to attract test-takers should be revised or dropped (Kehoe, 1995), because an MC item without effective distractors would be less discriminating (Haladyna & Downing, 1993). Thus, distractor analysis is necessary to ensure the measurement quality of MC items.Many methods can be used to detect implausible distractors (Gierl et al., 2017). An intuitive strategy is to observe the frequency of options used by test-takers. Specifically, practitioners can easily flag nonfunctioning distractors that are infrequently (e.g., <5%) used in a test (Rao et al., 2016;Tarrant et al., 2009). Additionally, to examine the discrimination of distractors, researchers have also calculated the mean score of test-takers who choose a distractor (Haladyna & Rodriguez, 2013) or the point-biserial correlation for a distractor (Attali & Fraenkel, 2000). Alternatively,

show abstract

A Mixture IRTree Model for Performance Decline and Nonignorable Missing Data

Cited by 14 publications

References 62 publications

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

On the Treatment of Missing Item Responses in Educational Large-Scale Assessment Data: An Illustrative Simulation Study and a Case Study Using PISA 2018 Mathematics Data

Dynamic Response Strategies: Accounting for Response Process Heterogeneity in IRTree Decision Nodes

Exploring the Impact of Random Guessing in Distractor Analysis

Contact Info

Product

Resources

About