The Empirical Power and Type I Error Rates of the GBT and ω Indices in Detecting Answer Copying on Multiple-Choice Tests

Abstract

The generalized binomial test (GBT) and ω indices are the most recent methods suggested in the literature to detect answer copying behavior on multiple-choice tests. The ω index is one of the most studied indices, but there has not yet been a systematic simulation study for the GBT index. In addition, the effect of the ability levels of the examinees in answer copying pairs on the statistical properties of the GBT and ω indices have not been systematically addressed as yet. The current study simulated 500 answer copying pairs for each of 1,440 conditions (12 source ability level × 12 cheater ability level × 10 amount of copying) to study the empirical power and 10,000 pairs of independent response vectors for each of 144 conditions (12 source ability level × 12 cheater ability level) to study the empirical Type I error rates of the GBT and ω indices. Results indicate that neither GBT nor ω inflated the Type I error rates, and they are reliable to use in practice. The difference in statistical power of these two methods was very small, and GBT performs slightly better than does ω. The main effect for the amount of copying and the interaction effect between source ability level and the amount of copying are found to be very strong while all other main and interactions effects are negligible.

Effects of Test Level Discrimination and Difficulty on Answer-Copying Indices

“…Random item copying procedure was applied for fixed copying amount of 40% for each replication. Amount of copying was taken and fixed as 40% for each condition because previous studies [23], [26] [30], [31] showed that power rates of indices were quite satisfactory for 30% and more.…”

“…Ability levels of copier and source are important for the performance of answer-copying indices. Several studies in the literature have pointed to the importance of examinee ability levels on answer copying [26], [34]. For this study the abilities of examinees were obtained by using irtoys [35] R package depending on 2PL Item Response Model.…”

“…Numerous studies have been conducted to compare answer-copying detection indices for several conditions in terms of Type I error and power studies [5]- [7], [12], [13], [15], [19], [20], [23], [25], [26], [28], [30]- [33]. Literature review points that two of the answer copying indices, ω [23] and GBT [28], are remarkable with low Type I error rates and high detection power ratios [19], [20], [23], [25], [26], [30]- [34]. However there are limited studies which compare the performance of these indices in literature.…”

“…However there are limited studies which compare the performance of these indices in literature. Zopluoglu and Davenport [26] investigated the performance of ω and GBT for several amount of copying and copier & source ability crossing. Zopluoglu [25] investigated the performance of ω, GBT, K, , , S 1 , S 2 on several simulated data sets and real data set.…”

“…The expected values of each response alternative will be estimated by using nominal response model estimations under the restriction of ability parameters of probable copier and source. Despite the fact that ω's original formula was developed by using Nominal Response Model and using option matrix of a multiple choice item test, it is also possible to compute ω on dichotomous data sets by using 1PL, 2PL or 3PL models of IRT [25], [26]. Literature review presents several studies which compare ω for nominal responses and dichotomous responses.…”