Effectiveness of Equating at the Passing Score for Exams With Small Sample Sizes

Abstract

This article explores the amount of equating error at a passing score when equating scores from exams with small samples sizes. This article focuses on equating using classical test theory methods of Tucker linear, Levine linear, frequency estimation, and chained equipercentile equating. Both simulation and real data studies were used in the investigation. The results of the study supported past findings that as the sample sizes increase, the amount of bias in the equating at the passing score decreases. The research also highlights the importance for practitioners to understand the data, to have an informed expectation of the results, and to have a documented rationale for an acceptable amount of equating error.An examinee enters a testing center and scores a 45 out of 80 on Form A of an exam. Another examinee of the same ability completes a parallel form, Form B, of the exam and scores a 43 out of 80 on the exam. Form B is statistically equated to Form A. Based on the information provided thus far, Form B is seemingly more difficult than Form A. However, if additional information was provided that stated only 10 examinees completed each of the two forms, would you still believe that Form B is more difficult than Form A? How much confidence would you have in the interpretation of the equated Form B score? If the sample size increased to 25 on each form, how much more confidence would you have in the equated scores of Form B? How about 50 examinees on each form? 125? 250? At what point is the sample size for the equating great enough that you have enough confidence in interpreting the results? The likely answer is: It depends.There are multiple factors that play into the role of deciding when to equate two forms and determining the amount of acceptable equating error. One of the main factors is the purpose of the equating. Is the equating being performed for a low, medium, or high stakes exam? A program may tolerate more equating error in a low stakes exam than a high stakes exam. However, if two exam forms differ in difficulty and the sample size is small, what action should be taken? Should some form of equating be performed or should no equating be performed?The purpose of this article is to discuss the amount of error in equating small sample sizes at the passing score for credentialing exams. This differs from previous research on equating with small sample sizes because most such research focuses on equating accuracy across the entire range of possible scores. The goal of this article is to determine if equating two forms using methods normally recommended for higher sample sizes would effectively work for smaller sample sizes if the examinees' population distribution is centered near the passing score.

Empirical ensemble equating under the NEAT design inspired by machine learning ideology

“…Each condition was replicated 100 times. Three measures were used according to the literature (i.e., Wolkowitz & Wright, 2019;Zeng, 1993) where SS was the sample size, and x . y p was the equated score of an individual examinee.…”

The NEAT Equating Via Chaining Random Forests in the Context of Small Sample Sizes: A Machine-Learning Method

“…Step 5: Steps 2 to 4 were repeated 100 times. Two measures were used according to the literature (i.e., Wolkowitz & Wright, 2019; Zeng, 1993)—the average absolute bias (BIAS) and root mean square difference (RMSD):…”

A Method to Increase the Power of Monte Carlo Method: Increasing the Number of Iteration

“…Out of these, 13 have used simulated data and only seven of them provide information regarding the number of iteration. In these studies, 100 (Svetina, Ling Liaw, and Rutkowski, 2019;Wind and Jones, 2019;Zhang, Wang, and Shi, 2019) and 200 (Wolkowitz, 2019) it has been reported to have generated data through iteration. In literature, there are studies using different iteration number such as 10000 (Saeki ve Tango, 2014), 1000 (Kannan, Sgammato, Tannenbaum and Katz 2015;Bionis, Huang and Gramacy, 2019), 500 (Glen Satten, Flanders and Yang, 2001) 100 (Fay and Gerow;2013), 10 (Kéry andRoyle, 2016;Murie and Nadon, 2018).…”