Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items

Abstract

In educational and psychological measurement, researchers and/or practitioners are often interested in examining whether the ability of an examinee is the same over two sets of items. Such problems can arise in measurement of change, detection of cheating on unproctored tests, erasure analysis, detection of item preknowledge, etc. Traditional frequentist approaches that are used in such problems include the Wald test, the likelihood ratio test, and the score test (e.g., Fischer, Appl Psychol Meas 27:3-26, 2003; Finkelman, Weiss, & Kim-Kang, Appl Psychol Meas 34:238-254, 2010; Glas & Dagohoy, Psychometrika 72:159-180, 2007; Guo & Drasgow, Int J Sel Assess 18:351-364, 2010; Klauer & Rettig, Br J Math Stat Psychol 43:193-206, 1990; Sinharay, J Educ Behav Stat 42:46-68, 2017). This paper shows that approaches based on higher-order asymptotics (e.g., Barndorff-Nielsen & Cox, Inference and asymptotics. Springer, London, 1994; Ghosh, Higher order asymptotics. Institute of Mathematical Statistics, Hayward, 1994) can also be used to test for the equality of the examinee ability over two sets of items. The modified signed likelihood ratio test (e.g., Barndorff-Nielsen, Biometrika 73:307-322, 1986) and the Lugannani-Rice approximation (Lugannani & Rice, Adv Appl Prob 12:475-490, 1980), both of which are based on higher-order asymptotics, are shown to provide some improvement over the traditional frequentist approaches in three simulations. Two real data examples are also provided.

The Lack of Robustness of a Statistic Based on the Neyman–Pearson Lemma to Violations of Its Underlying Assumptions

“…Because the null and alternative hypotheses underlying the SLR statistic are not simple hypotheses, the SLR statistic cannot be expected to be the most powerful statistic for detecting preknowledge on a known set of compromised items, even though the statistic is based on a likelihood ratio. However, Sinharay (2017c), Sinharay (2017d), Sinharay and Jensen (2019), and Wang et al (2019) found the SLR statistic to be as powerful as or more powerful than the existing statistics in detecting item preknowledge and found the Type I error rate of the statistic to be very close to the nominal level. Specifically, Sinharay (2017d) proved the performance of the SLR statistic to be very similar to that of the posterior shift statistic that was found to be the most powerful among eight preknowledge-detection statistics by Belov (2016).…”

The Use of the Posterior Probability in Score Differencing

“…A large value of L S leads to the rejection of the null hypothesis. Sinharay (2017aSinharay ( , 2017b and Wang et al (2019) demonstrated using real and simulated data that the performance of L S was satisfactory compared to that of several existing statistics for detecting item preknowledge, and Sinharay and Jensen (2019) found L S to have satisfactory Type I error rates and power in several applications of score differencing. Therefore, L S is the only frequentist statistic for score differencing that is considered in this article.…”

“…The source of the data set is Cizek and Wollack (2017, p. 14). Researchers such as Sinharay (2017a), Jensen (2019), andZopluoglu (2017) analyzed the same data set to detect various types of test fraud. The test form comprises 170 operational items that are dichotomously scored.…”

“…In addition, 48 examinees were flagged by the organization as possible cheaters from a variety of statistical analysis and a rigorous investigative process that brought in other information; given the rigor of the investigative process, it is reasonable to treat these examinees as true cheaters. As in Sinharay (2017a) and Sinharay and Jensen (2019), the interest here is in detecting item preknowledge, that is, detecting the examinees who may have benefited from the preknowledge of the 61 compromised items.…”

“…One of the categories is "score differencing," which involves a test of the null hypothesis of equal ability of an examinee over two sets of items I 1 and I 2 . Score differencing can be used to detect several types of test fraud including item preknowledge (e.g., Sinharay, 2017aSinharay, , 2017bSinharay & Jensen, 2019), fraudulent erasures (e.g., Sinharay et al, 2017), fraudulent gain scores (e.g., Fischer, 2003), and cheating on unproctored tests (e.g., Guo & Drasgow, 2010).…”