Commentary: On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

Pohl, Steffi; Davier, Matthias von

doi:10.3389/fpsyg.2018.01988

Cited by 8 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ultimate goal is to estimate students’ target latent ability accurately by properly accounting for their test‐taking strategies. A key assumption of our model, consistent with prior literature (e.g., Pohl et al., 2019; Pohl & von Davier, 2018; van der Linden, 2007), is that an examinee shows effective ability at a chosen (effective) speed for timed tests. The same assumption is also adopted by Lee and Ying (2015), in which they proposed a mixture cure‐rate model to distinguish incorrect answers due to either low proficiency or lack of time.…”

Section: Introductionsupporting

confidence: 71%

A Response Time Process Model for Not‐Reached and Omitted Items

Wang

2020

J Educational Measurement

View full text Add to dashboard Cite

Item nonresponses are prevalent in standardized testing. They happen either when students fail to reach the end of a test due to a time limit or quitting, or when students choose to omit some items strategically. Oftentimes, item nonresponses are nonrandom, and hence, the missing data mechanism needs to be properly modeled. In this paper, we proposed to use an innovative item response time model as a cohesive missing data model to account for the two most common item nonresponses: not-reached items and omitted items. In particular, the new model builds on a behavior process interpretation: a person chooses to skip an item if the required effort exceeds the implicit time the person allocates to the item (Lee & Ying, 2015; Wolf, Smith, & Birnbaum, 1995), whereas a person fails to reach the end of the test due to lack of time. This assumption was verified by analyzing the 2015 PISA computerbased mathematics data. Simulation studies were conducted to further evaluate the performance of the proposed Bayesian estimation algorithm for the new model and to compare the new model with a recently proposed "speed-accuracy + omission" model (Ulitzsch, von Davier, & Pohl, 2019). Results revealed that all model parameters could recover properly, and inadequately accounting for missing data caused biased item and person parameter estimates.

show abstract

Section: Introductionsupporting

confidence: 71%

A Response Time Process Model for Not‐Reached and Omitted Items

Wang

2020

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…However, even test takers that are within the time limit differ in their speed. Thus, different target abilities would be estimated for both groups (see also Pohl & von Davier, 2018). This is not the case in the SA model.…”

Section: Implications For the Practice Of Dealing With Not-reached Items Due To Time Limitsmentioning

confidence: 99%

“…We suggest describing the performance of groups of test takers (for example grouped by language, country, or school type) by all aspects of performance: ability and speed (and/or missing propensity in case of other reasons for missing values) and use all of these for evaluating the performance (see also Pohl & von Davier, 2018). This allows to develop a richer description of differences in performance and to disentangle the different constructs involved.…”

Section: Implications For the Practice Of Dealing With Not-reached Items Due To Time Limitsmentioning

confidence: 99%

See 1 more Smart Citation

Using Response Times to Model Not-Reached Items due to Time Limits

2019

Self Cite

View full text Add to dashboard Cite

Missing values at the end of a test typically are the result of test takers running out of time and can as such be understood by studying test takers' working speed. As testing moves to computer-based assessment, response times become available allowing to simulatenously model speed and ability. Integrating research on response time modeling with research on modeling missing responses, we propose using response times to model missing values due to time limits. We identify similarities between approaches used to account for not-reached items (Rose et al. in ETS Res Rep Ser 2010:i-53, 2010 and the speed-accuracy (SA) model for joint modeling of effective speed and effective ability as proposed by van der Linden (Psychometrika 72(3): 2007). In a simulation, we show (a) that the SA model can recover parameters in the presence of missing values due to time limits and (b) that the response time model, using item-level timing information rather than a count of not-reached items, results in person parameter estimates that differ from missing data IRT models applied to not-reached items. We propose using the SA model to model the missing data process and to use both, ability and speed, to describe the performance of test takers. We illustrate the application of the model in an empirical analysis.

show abstract

“…It is important to note that the validity of our conclusions from the simulation study for real-life settings heavily depends on whether the data-generating model used in the simulation mirrors the empirical situation. Which missing data approach is most appropriate depends on (a) which target ability is to be estimated and (b) which missing data mechanism is assumed (Pohl & von Davier, 2018; Tijmstra & Bolsinova, 2018). We take the view here of aiming to estimate effective ability given the chosen speed (van der Linden, 2007) and the chosen percentage of missing values.…”

Section: Discussionmentioning

confidence: 99%

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Sachse

Mahler

Pohl

2019

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms.

show abstract

Commentary: On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items

Cited by 8 publications

References 7 publications

A Response Time Process Model for Not‐Reached and Omitted Items

A Response Time Process Model for Not‐Reached and Omitted Items

Using Response Times to Model Not-Reached Items due to Time Limits

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Contact Info

Product

Resources

About