Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

Linden, Willem J. van der; Guo, Fanmin

doi:10.1007/s11336-007-9046-8

Cited by 140 publications

(179 citation statements)

References 20 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…Residual analysis and checks on aberrant response patterns have been proposed, and extensive literature reviews have been done by Sijtsma (1995, 2001) and Karabatsos (2003). A check for RT patterns has been discussed by van der Linden and van Krimpen-Stoop (2003) and van der Linden and Guo (2008), who have mainly been interested in detecting cheating behavior. Their method is based on evaluating the posterior probability that an observed RT is lower or higher than the posterior predicted RT under the model.…”

Section: Discussionmentioning

confidence: 99%

“…Van der Linden and Guo (2008) introduce two types of aberrant response behavior: (a) attempts at memorization, which might reveal themselves by random RTs; and (b) item preknowledge, which might result in an unusual combination of a correct response and RTs. RT patterns are considered to be suspicious when an answer is correct and the RT is relatively small while the probability of success on the item is low.…”

Section: Introductionmentioning

confidence: 99%

“…Based on this log-normal RT model, a general approach to detect aberrant response behavior can be considered in which checks can be used to flag respondents or items that need further consideration. Van der Linden and Guo (2008) already indicated that test takers may show aberrant behavior for several reasons, and it would be wrong to jump to conclusions. Checks could be used routinely in order to flag test takers or items that may need further consideration or to support observations by proctors or other evidence.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Testing for Aberrant Behavior in Response Time Modeling

Marianti

Fox

Avetisyan

et al. 2014

Journal of Educational and Behavioral Statistics

101

View full text Add to dashboard Cite

Many standardized tests are now administered via computer rather than paperand-pencil format. In a computer-based testing environment, it is possible to record not only the test taker's response to each question (item) but also the amount of time spent by the test taker in considering and answering each item. Response times (RTs) provide information not only about the test taker's ability and response behavior but also about item and test characteristics. This study focuses on the use of RTs to detect aberrant test-taker responses. An example of such aberrance is a correct answer with a short RT on a difficult question. Such aberrance may be displayed when a test taker or test takers have preknowledge of the items. Another example is rapid guessing, wherein the test taker displays unusually short RTs for a series of items. When rapid guessing occurs at the end of a timed test, it often indicates that the test taker has run out of time before completing the test. In this study, Bayesian tests of significance for detecting various types of aberrant RT patterns are proposed and evaluated. In a simulation study, the tests were successful in identifying aberrant response patterns. A real data example is given to illustrate the use of the proposed person-fit tests for RTs.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Testing for Aberrant Behavior in Response Time Modeling

Marianti

Fox

Avetisyan

et al. 2014

Journal of Educational and Behavioral Statistics

101

View full text Add to dashboard Cite

show abstract

“…To evaluate the fit of the model to the RT data we used Bayesian residual analysis (Klein Entink et al, 2009;van der Linden & Guo, 2008). The left-tail posterior predictive probability for each observed log-RT (i.e., the probability of observing a value smaller than the observed ln T pi ) was approximated by …”

Section: Analysis Of the Dutch Language Test Datamentioning

confidence: 99%

Improving precision of ability estimation: Getting more from response times

Bolsinova

Tijmstra

2017

Brit J Math & Statis

View full text Add to dashboard Cite

By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross-loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA-only model is considered. The model is applied to data from a high-stakes educational test.

show abstract

“…Residual analysis and person-fit statistics have been proposed to examine aberrant response patterns, or more specially, item preknowledge (e.g., Drasgow, Levine, & Williams, 1985;Karabatsos, 2003;McLeod & Lewis, 1999;McLeod, Lewis, & Thissen, 2003;Meijer & Sijtsma, 2001;Segall, 2002;Shu, Henson, & Luecht, 2013). Residual analysis of both response and timing data has also been considered for the same purpose (e.g., van der Linden & Guo, 2008). More recently, there has been research concerning test collusion, or large-scale sharing of test materials or answers to one or more subsets of items prior to the examination (Belov, 2013(Belov, , 2014Wollack & Maynes, 2016;Zhang, Searcy, & Horn, 2011).…”

mentioning

confidence: 99%

A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses

Haberman

Lee

2017

ETS Research Report Series

View full text Add to dashboard Cite

In investigations of unusual testing behavior, a common question is whether a specific pattern of responses occurs unusually often within a group of examinees. In many current tests, modern communication techniques can permit quite large numbers of examinees to share keys, or common response patterns, to the entire test. To address this issue, statistical methods are provided to identify examinees in a test with answers that exactly match and to assess whether such exact matches are unusual. In addition, methodology is provided to identify examinees with response patterns unusually similar to circulated keys. Application is made to a testing program.

show abstract

Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

Abstract: adaptive testing, Bayesian predictive checks, cheating, collateral information, hierarchical modeling, response times,

Cited by 140 publications

References 20 publications

Testing for Aberrant Behavior in Response Time Modeling

Testing for Aberrant Behavior in Response Time Modeling

Improving precision of ability estimation: Getting more from response times

A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses

Contact Info

Product

Resources

About