The Relationship between Response-Time Effort and Accuracy in PISA Science Multiple Choice Items

A New Method for Setting Response-Time Threshold to Detect Test Takers’ Rapid Guessing Behavior

“…This phenomenon often occurs in analysing response time, such as reported by Sahin and Colvin [21]. They found one item with a bimodal distribution pattern before the median response time [23]. Although the VI method is most widely used [18,19], the fundamental problem with this method is related to the bimodal distribution that is rarely founding in many items [26].…”

“…In the data that varies between responses, the median is robust than other methods, let alone a large number of samples [21]. Sahin and Colvin [22] and Michaelides et al [23] used the median in analysing the bimodal distribution pattern of response time.…”

Random Responders in the TIMSS 2015 Student Questionnaire: A Threat to Validity?

“…In some cases, the differences in response behavior also lead to large group differences in the scale‐related impact statistics (i.e., internal scale statistics, correlations across scales, and correlations with achievement). It has especially been these differences between the actual responder groups (e.g., (un)motivated; (in)sufficient effort groups) that have been of particular interest in the context of international large‐scale educational assessments (for examples in terms of average achievement scores, see e.g., Eklöf et al., 2014; Hopfenbeck & Kjærnsli, 2016 or average accuracy scores, see e.g., Michaelides, Ivanova, & Nicolaou, 2020). Because the random responder group by definition tends to score near the survey scale average, larger group differences on the relevant impact statistics coincide with more homogeneous outspoken responses (i.e., far from the theoretical scale average) by the regular group.…”

“…However, it seems natural that a more structured data quality monitoring process would be centrally organized or at least facilitated. To the organizing parties and stakeholders of the international large‐scale educational assessments, we would therefore make a plea for the default inclusion of proven standard survey measures to facilitate detection of invalid response behavior (e.g., Breitsohl & Steidelmüller, 2018; Leiner, 2019): (i) the inclusion of an instructed response item (e.g., “Please mark slightly agree”) or bogus item (e.g., “I have never used a computer”) at a few random moments throughout the survey for an individual pupil in combination with a warning at the start of the survey that such items can be included, and (ii) the provision of individual survey completion speed indicators to help track rushed responding (cf., analog to rapid guessing in the achievement context, e.g., Michaelides, Ivanova, & Nicolaou, 2020; Wise et al., 2020) with the ethical requirement that participants are informed about such data being collected. The recent move to computer‐based assessment, would make it straightforward to implement both measures.…”

On Joining a Signal Detection Choice Model with Response Time Models

“…This is analogous to the probability of being in the slower state for any given item, given by 1-λ ν in the mixture lognormal RT model. Note that figure 4 of Michaelides et al (2020), with the x-axis reflected ("1" in their figure 4 is "0" in the current Figure 2), has the same shape as the distribution of λ ν found here and shown in Figure 2-both distributions show a left mode and high positive skew, and in this way the current results are consistent with those found by Michaelides et al They also found that accuracy was lower in the higher speed state, which also agrees with the current results-note that accuracy is lower in the "don't know" state of the SDT model because detection is zero (δd j = 0), and so if being in the higher speed state is associated with being in the "don't know" state, as found here, then accuracy will tend to be lower, in agreement with Michaelides et al's results. Thus, results for the SDT/mixRT model are consistent with those found by other means, with the advantage that a model-based approach offers a more detailed and unified analysis.…”