Modeling NAEP Test-Taking Behavior Using Educational Process Analysis

Patel, Nirmal; Sharma, Aditya; Shah, Tirth; Lomas, Derek

doi:10.5281/zenodo.5275312

Cited by 2 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In line with Hypothesis 1, the extended DLC-TL IRT model that included the new indicators showed a better fit as compared with a model with response times alone, supporting the finding of previous studies that process data beyond response time might indeed provide valuable information on response engagement ( Ivanova et al, 2020 ; Lundgren & Eklöf, 2020 ; Patel et al, 2021 ; Sahin & Colvin, 2020 ). However, a closer inspection of the results revealed that only item response times and text reread predicted response engagement while answer change and item revisit did not.…”

Section: Discussionsupporting

confidence: 84%

“… Ivanova and colleagues (2020) found that the number of actions in constructed response items was related to test performance, self-reported test-taking effort, and item position. Finally, Patel and colleagues (2021) used a machine learning approach to predict test-taking efficiency in a test of mixed item types. Besides measures related to test or item response time, the number of certain actions (e.g., answer changes and navigation button use) were identified as important features of test efficiency.…”

Section: Enhancing the Identification Of Disengaged Responses With Pr...mentioning

confidence: 99%

“…One potential source of information on test-taking behavior that has been increasingly available with the advent of computer-based assessments is process data, which provides a rich array of easily accessible information on the test-taking process. Few studies have yet investigated the potential of process data beyond item response times for the identification of disengaged responding ( Ivanova et al, 2020 ; Lundgren & Eklöf, 2020 ; Patel et al, 2021 ; Sahin & Colvin, 2020 ). So far, available studies either concentrated on response engagement in interactive items or were rather exploratory.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Welling

Gnambs

Carstensen

2023

Educational and Psychological Measurement

View full text Add to dashboard Cite

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

show abstract

Section: Discussionsupporting

confidence: 84%

Section: Enhancing the Identification Of Disengaged Responses With Pr...mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Welling

Gnambs

Carstensen

2023

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…In line with Hypothesis 1, the extended DLC-TL IRT model that included the new indicators showed a better fit as compared to a model with response times alone, supporting the finding of previous studies that process data beyond response time might indeed provide valuable information on response engagement (Ivanova et al, 2020;Lundgren & Eklöf, 2020;Patel et al, 2021;Sahin & Colvin, 2020). However, a closer inspection of the results revealed that only item response times and text reread predicted response engagement while answer change and item revisit did not.…”

Section: Discussionsupporting

confidence: 84%

“…One potential source of information on test-taking behavior that has been increasingly available with the advent of computer-based assessments is process data, which provides a rich array of easily accessible information on the test-taking process. Few studies have yet investigated the potential of process data beyond item response times for the identification of disengaged responding (Lundgren & Eklöf, 2020;Ivanova et al, 2020;Patel et al, 2021;Sahin & Colvin, 2020). So far, available studies either concentrated on response engagement in interactive items or were rather exploratory.…”

Section: Identifying Disengaged Responding In Multiple-choice Items: ...mentioning

confidence: 99%

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Welling¹,

Gnambs²,

Carstensen³

2023

Preprint

View full text Add to dashboard Cite

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In the present study, three process data variables – text reread, item revisit, and answer change – were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class-item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

show abstract

Modeling NAEP Test-Taking Behavior Using Educational Process Analysis

Cited by 2 publications

References 25 publications

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model with Novel Process Data Indicators

Contact Info

Product

Resources

About