Testing for Aberrant Behavior in Response Time Modeling

Abstract

Many standardized tests are now administered via computer rather than paperand-pencil format. In a computer-based testing environment, it is possible to record not only the test taker's response to each question (item) but also the amount of time spent by the test taker in considering and answering each item. Response times (RTs) provide information not only about the test taker's ability and response behavior but also about item and test characteristics. This study focuses on the use of RTs to detect aberrant test-taker responses. An example of such aberrance is a correct answer with a short RT on a difficult question. Such aberrance may be displayed when a test taker or test takers have preknowledge of the items. Another example is rapid guessing, wherein the test taker displays unusually short RTs for a series of items. When rapid guessing occurs at the end of a timed test, it often indicates that the test taker has run out of time before completing the test. In this study, Bayesian tests of significance for detecting various types of aberrant RT patterns are proposed and evaluated. In a simulation study, the tests were successful in identifying aberrant response patterns. A real data example is given to illustrate the use of the proposed person-fit tests for RTs.

Improving precision of ability estimation: Getting more from response times

“…When this assumption does not hold for a particular response, extreme RT could be a consequence which in turn would distort estimated speed and the residuals obtained. This means that when there is a risk of such aberrant responding behaviour, using techniques to detect aberrant responses is advisable (see, for example, Marianti, Fox, Avetisyan, Veldkamp, & Tijmstra, 2014;van der Linden & Guo, 2008). k**, along with other model parameters).…”

Joint Modeling of Ability and Differential Speed Using Responses and Response Times

“…More generally, potential threats to the validity of the test scores (e.g., guessing, cheating) can be evaluated by exploring the speed trajectories of test takers in relation to ability. Statistical tests similar to the tests for aberrant speed behavior reported by Marianti et al (2014) could be developed to identify extreme changes in speed behavior.…”

“…Changes in time management could be required to finish the test in time, or test takers could decide to work slower to improve their level of accuracy. Working speed can also vary when test takers show aberrant response behavior, such as cheating or guessing (Marianti, Fox, Avetisyan, Veldkamp, & Tijmstra, 2014).…”

Stochastic programming for individualized test assembly with mixture response time models

“…In that case, it might pay to administer a larger number of items with shorter RTs that together provide more information than the most informative item. RTs also reveal which items are more sensitive to working speed than others (Marianti, Fox, Avetisyan, & Veldkamp, 2014). In order to prevent differential speededness in high-stakes testing situations, this sensitivity of items to working speed is very useful in the test development process.…”