Variation in Respondent Speed and its Implications: Evidence from an Adaptive Testing Scenario

Abstract

As our ability to collect data about respondents increases, approaches for incorporating ancillary data features such as response time are of heightened interest. Models for response time have been advanced, but relatively limited large-scale empirical investigations have been conducted. We take advantage of a unique and massive dataset—data from computer adaptive administrations of the NWEA MAP Growth assessment in two states consisting of roughly 1/4 billion item responses—containing both item responses plus response times to shed light on emergent features of response time behavior. We focus on two behaviors in particular. The first, response acceleration, is a reduction in response time for responses that occur relatively late on the assessment. We further note that such reductions are heterogeneous as a function of estimated ability (lower ability estimates are associated with larger increases in acceleration) and that reductions in response time on later items lead to reductions in accuracy relative to expectation. We also document variation in interplay between speed and accuracy. In some cases, additional time spent on an item is associated with an increase in accuracy; in other cases, the opposite is true. This finding has potential connections to the nascent literature on different within-person response processes. We argue that our approach may be useful in other settings and that the behaviors observed here should be of interest in other data.

Speed–Accuracy Trade-Off? Not So Fast: Marginal Changes in Speed Have Inconsistent Relationships With Accuracy in Real-World Settings

“…Other features of data collection may be relevant. We have not addressed ordering effects (Debeer & Janssen, 2013; Domingue et al, 2021; Vida et al, 2021). There are presumably motivational differences across the data sets that we do not measure and cannot study.…”

Speed accuracy tradeoff? Not so fast: Marginal changes in speed have inconsistent relationships with accuracy in real-world settings

“…The approach used here-in particular, combining 206 probabilities from item response models with fixed 207 effects-draws from earlier work [19]. 3 We have also 208 verified that it behaves as expected via simulation in 209 the context of several different models for the joint 210 distribution of time and accuracy, see SI.…”

“…In many cases, items later in the test may appear harder than they would if presented earlier in the test. This may be due, in part, to systematic changes in response time devoted to such items [19].…”

Investigating the Effect of Differential Rapid Guessing on Population Invariance in Equating

“…Prior research has shown that examinees with lower ability are more likely to engage in RG, particularly as a test progresses (e.g., Deribo et al, 2021; Domingue et al, 2021; Rios et al, 2017). To investigate the influence of this factor, two conditions were manipulated to reflect differences in the ability characteristics of rapid guessers.…”

“…3This RG generating process also reflects response acceleration (i.e., response time reduction for later responses provided in a test) observed in prior applied analyses (Domingue et al, 2021). …”