Detecting Fraudulent Erasures at an Aggregate Level

Abstract

Wollack, Cohen, and Eckerly suggested the “erasure detection index” (EDI) to detect fraudulent erasures for individual examinees. Wollack and Eckerly extended the EDI to detect fraudulent erasures at the group level. The EDI at the group level was found to be slightly conservative. This article suggests two modifications of the EDI for the group level. The asymptotic null distribution of the two modified indices is proved to be the standard normal distribution. In a simulation study, the modified indices are shown to have Type I error rates close to the nominal level and larger power than the index of Wollack and Eckerly. A real data example is also included.

Monitoring Item Performance With CUSUM Statistics in Continuous Testing

“…As mentioned earlier, the SE level of 0.25 was set to represent the high-stakes admission test we studied (reliability equal to 0.94). The impact of ability estimation error is expected to become more serious when applying the CUSUM procedure to less reliable tests, and some modifications to the proposed CUSUM procedure may be needed to adjust for this source of errors for these tests (see, e.g., Sinharay, 2018, for a possible adjustment approach).…”

The Use of Theory of Linear Mixed-Effects Models to Detect Fraudulent Erasures at an Aggregate Level

“…Let μ gk and σ gk , respectively, denote the expectation and standard deviation of X gk given the true ability parameter ( θ gk ) of the corresponding examinee. The ability θ gk is unknown and is estimated from the responses on the nonerased items, that is, items in E gk c (Sinharay, 2018; Wollack et al, 2015). The estimates of μ gk and σ gk are denoted as μ ^ gk and σ ^ gk , respectively, and are obtained by replacing ...…”

“…Wollack and Eckerly (2017) and Sinharay (2018) extended the index of Wollack et al (2015) to suggest three EDIs that can be used to perform erasure analysis at an aggregate or group level, where the groups could refer to the classes, schools, or districts that the examinees belong to. The main goal of this article is to continue the research of Wollack and Eckerly (2017) and Sinharay (2018) and to propose a new EDI for performing erasure analysis at an aggregate level using the theory of linear mixed-effects models (LMMs; e.g., McCulloch et al, 2008). It is demonstrated that the new EDI has larger power than the indices of Wollack and Eckerly (2017) and Sinharay (2018).…”

“…The main goal of this article is to continue the research of Wollack and Eckerly (2017) and Sinharay (2018) and to propose a new EDI for performing erasure analysis at an aggregate level using the theory of linear mixed-effects models (LMMs; e.g., McCulloch et al, 2008). It is demonstrated that the new EDI has larger power than the indices of Wollack and Eckerly (2017) and Sinharay (2018). In addition, the new EDI is shown to have satisfactory Type I error rates in all simulation conditions.…”

“…The next section includes reviews of the EDIs that were suggested by Wollack et al (2015), Wollack and Eckerly (2017), and Sinharay (2018) to detect fraudulent erasures. The new EDI, which is based on the theory of LMMs, is introduced in the Method section.…”