Improving the reliability of student scores from speeded assessments: an illustration of conditional item response theory using a computer-administered measure of vocabulary

Abstract

A growing body of literature suggests that response latency, the amount of time it takes an individual to respond to an item, may be an important factor to consider when using assessment data to estimate the ability of an individual. Considering that tests of passage and list fluency are being adapted to a computer administration format, it is possible that accounting for individual differences in response times may be an increasingly feasible option to strengthen the precision of individual scores. The present research evaluated the differential reliability of scores when using classical test theory and item response theory as compared to a conditional item response model which includes response time as an item parameter. Results indicated that the precision of student ability scores increased by an average of 5 % when using the conditional item response model, with greater improvements for those who were average or high ability. Implications for measurement models of speeded assessments are discussed.

The Effects of Log Data on Students’ Performance

“…However, none were found to have a significant effect on student performance, and these results are likely to be related to previous findings. Given the non-uniform distribution of RTs among items and students, RTs of students having a higher reading ability or ICT competency would also have a Petscher, Mitchell, and Foorman (2015). In the study by Petscher et al (2015), the variability of RTs of students having higher reading ability showed more functional information compared to students with lower or moderate ability.…”

“…Given the non-uniform distribution of RTs among items and students, RTs of students having a higher reading ability or ICT competency would also have a Petscher, Mitchell, and Foorman (2015). In the study by Petscher et al (2015), the variability of RTs of students having higher reading ability showed more functional information compared to students with lower or moderate ability. On the contrary, Su and Davison (2019) found that students with high reading ability had lower RTs while answering the items correctly.…”

Task Characteristics as Source of Difficulty and Moderators of the Effect of Time-on-Task in Digital Problem-Solving

“…Moreover, the positive effect of time-on-task was shown to become stronger in the case of harder non-routine tasks (Goldhammer et al., 2014). However, task accuracy is not always positively linked to the amount of time individuals are engaged with the task (Goldhammer et al., 2013; Goldhammer, Naumann, et al., 2017; Petscher et al., 2015; Scherer et al., 2015; Vörös & Rouet, 2016). Higher ability test-takers was shown to spend less time on reading tasks and correctly accomplish reading tasks in less time than lower ability test-takers (Petscher et al., 2015; Su & Davison, 2019).…”

“…However, task accuracy is not always positively linked to the amount of time individuals are engaged with the task (Goldhammer et al., 2013; Goldhammer, Naumann, et al., 2017; Petscher et al., 2015; Scherer et al., 2015; Vörös & Rouet, 2016). Higher ability test-takers was shown to spend less time on reading tasks and correctly accomplish reading tasks in less time than lower ability test-takers (Petscher et al., 2015; Su & Davison, 2019). Finally, it was proposed that the link between time-on-task and accuracy depends on the quantity of routinized sub-tasks of problem-solving processes (Goldhammer, Naumann et al., 2017; Naumann & Goldhammer, 2017).…”

When a silent reading fluency test measures more than reading fluency: academic language features predict the test performance of students with a non-German home language

“…Furthermore, tablets and eye-tracking can be applied to measure the time spent on each sentence. A computer-based administration capturing both item response accuracy and response times would allow researchers to use conditional item response theory such as illustrated by Petscher, Mitchell, and Foorman (2015). This would shed more light on the demands that certain linguistic structures pose for (nonmonolingualhome) students, thus providing information that will be relevant for both research and practice.…”