How does the number of actions on constructed-response items relate to test-taking effort and performance?

Ivanova, Militsa; Michaelides, Michalis P.; Eklöf, Hanna

doi:10.1080/13803611.2021.1963939

Cited by 11 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In line with Hypothesis 1, the extended DLC-TL IRT model that included the new indicators showed a better fit as compared to a model with response times alone, supporting the finding of previous studies that process data beyond response time might indeed provide valuable information on response engagement (Ivanova et al, 2020;Lundgren & Eklöf, 2020;Patel et al, 2021;Sahin & Colvin, 2020). However, a closer inspection of the results revealed that only item response times and text reread predicted response engagement while answer change and item revisit did not.…”

Section: Discussionsupporting

confidence: 85%

“…One potential source of information on test-taking behavior that has been increasingly available with the advent of computer-based assessments is process data, which provides a rich array of easily accessible information on the test-taking process. Few studies have yet investigated the potential of process data beyond item response times for the identification of disengaged responding (Lundgren & Eklöf, 2020;Ivanova et al, 2020;Patel et al, 2021;Sahin & Colvin, 2020). So far, available studies either concentrated on response engagement in interactive items or were rather exploratory.…”

Section: Identifying Disengaged Responding In Multiple-choice Items: ...mentioning

confidence: 99%

See 1 more Smart Citation

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Welling¹,

Gnambs²,

Carstensen³

2023

Preprint

View full text Add to dashboard Cite

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In the present study, three process data variables – text reread, item revisit, and answer change – were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class-item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

show abstract

Section: Discussionsupporting

confidence: 85%

Section: Identifying Disengaged Responding In Multiple-choice Items: ...mentioning

confidence: 99%

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Welling¹,

Gnambs²,

Carstensen³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The use of response time data is one of the most established applications of response process data in assessment, with a degree of formalization about its methods (Lee & Jia, 2014; Li et al, 2017; Wise, 2019) and related discussion of validation (Goldhammer et al, 2021; Li et al, 2017; Reis Costa et al, 2021; Wise, 2017, 2019). The digital transition has witnessed extensive application, such as measures to support inferences about rapid guessing behaviors and disengagement in large-scale assessments (Ercikan et al, 2020; Goldhammer et al, 2017; Kroehne et al, 2020; Lundgren & Eklöf, 2020; Soland et al, 2018; Wise, 2017, 2019, 2020a, 2020b; Wise & Gao, 2017, Wise & Kong, 2005, Wise et al, 2019), and to explore the relationship between response times, accuracy and ability (e.g., Ivanova et al, 2020; Michaelides et al, 2020; Ranger et al, 2021; Reis Costa et al, 2021).…”

Section: Item Response Times and Disabilitymentioning

confidence: 99%

Process and Product in Computer-Based Assessments

Zumbo

Maddox

Care

2023

European Journal of Psychological Assessment

View full text Add to dashboard Cite

There is no consensus among assessment researchers about many of the central problems of response process data, including what is it and what is it comprised of. The Standards for Educational and Psychological Testing ( American Educational Research Association et al., 2014 ) locate process data within their five sources of validity evidence. However, we rarely see a conceptualization of response processes; rather, the focus is on the techniques and methods of assembling response process indices or statistical models. The method often overrides clear definitions, and, as a field, we may therefore conflate method and methodology – much like we have conflated validity and validation ( Zumbo, 2007 ). In this paper, we aim to clear the conceptual ground to explore the scope of a holistic framework for the validation of process and product. We review prominent conceptualizations of response processes and their sources and explore some fundamental questions: Should we make a theoretical and practical distinction between response processes and response data? To what extent do the uses of process data reflect the principles of deliberate, educational, and psychological measurement? To answer these questions, we consider the case of item response times and the potential for variation associated with disability and neurodiversity.

show abstract

“…Sahin and Colvin (2020) showed that the number and type of actions could in some cases slightly improve the response time thresholds in interactive tasks. Ivanova and colleagues (2020) found that the number of actions in constructed response items was related to test performance, self-reported test-taking effort, and item position. Finally, Patel and colleagues (2021) used a machine learning approach to predict test-taking efficiency in a test of mixed item types.…”

Section: Enhancing the Identification Of Disengaged Responses With Pr...mentioning

confidence: 99%

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Welling

Gnambs

Carstensen

2023

Educational and Psychological Measurement

View full text Add to dashboard Cite

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

show abstract

How does the number of actions on constructed-response items relate to test-taking effort and performance?

Cited by 11 publications

References 39 publications

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Process and Product in Computer-Based Assessments

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Contact Info

Product

Resources

About