Modeling Item Response Times With a Two‐State Mixture Model: A New Method of Measuring Speededness

Abstract

Speededness refers to the extent to which time limits affect examinees'test performance, and it is often measured by calculating the proportion of examinees who do not reach a certain percentage of test items. However, when tests are number‐right scored (i.e., no points are subtracted for incorrect responses), examinees are likely to rapidly guess on items rather than leave them blank. Therefore, this traditional measure of speededness probably underestimates the true amount of speededness on such tests. A more accurate assessment of speededness should also reflect the tendency of examinees to rapidly guess on items as time expires. This rapid‐guessing component of speededness can be estimated by modeling response times with a two‐state mixture model, as demonstrated with data from a computer‐ administered reasoning test. Taking into account the combined effect of unreached items and rapid guessing provides a more complete measure of speededness than has previously been available.

Joint Modeling of Ability and Differential Speed Using Responses and Response Times

“…Test takers may show solution behavior or rapid guessing behavior, where the guessed responses provide no information about the true achievement level of the test takers. Schnipke and Scrams (1997) considered the use of RTs to identify rapid guessing behavior for a speeded test. Wise and Kong (2005) also considered RTs to measure RT effort, which addresses the proportion of true solution behavior in contrast to rapid guessing behavior.…”

Reliability of Speeded Number‐right Multiple‐choice Tests

“…Because the research on this issue is very limited, it is difficult to determine the correctness of this two-state assumption. However, Schnipke and Scrams (1997) were successful in modeling item response times with such a two-state mixture model. The twostate assumption is plausible because examinees know in advance that the time limits of a particular test are very strict.…”

“…Following Schnipke and Scrams (1997) and Yamamoto (1995), it is assumed that examinees choose to engage in either solution behavior or rapid-guessing behavior in answering each item. Because examinees spend very little time on the item in rapid-guessing, their answers may be characterized as random guessing, and the probability of a correct answer for these responses can be assumed to be 1/k, where k is the number of options for the MC item.…”

Time to completion reveals problem-solving transfer

“…3). This observation is interesting since the sigmoid curve plays a prominent role in growth models [3] and time to completion of other psychological tasks [4,5] from sentence completion [6] to pigeon learning [7]. These curves are important for short term and long term objectives.…”

“…The only precedent we have found for the superiority of Ln(t) is in computer adapted testing [10] where a reanalysis of the data seems to favor this independent variable. In this context we note that it is for complex tasks like the problems given in our study but unlike the simple tasks in [4][5][6][7].…”