Evaluating the Impact of Releasing an Item Pool on a Test's Empirical Characteristics

Abstract

Protecting the security of examination questions is an important task for high-stakes examining boards/agencies and university programs. To maintain the security of questions, examining boards and university programs use a combination of prevention, detection, and enforcement strategies. A common prevention strategy is to establish a number of controls on access to questions; however, restricting access can motivate examinees to try harder to reconstruct questions that may appear on future versions of the test. Moreover, access to study materials by some groups and not others can present a challenge to the fairness of examinations. The aim of this study was to evaluate the empirical stability of test characteristics. Specifically, the primary research objective was to investigate the empirical stability of the items and test forms of a written examination before and after a specific policy decision was implemented. As a response to both of these concerns, this article describes a study that evaluated how psychometric (i.e., statistical) properties of test forms and individual questions might be affected by publicly releasing a larger number of questions from an item question pool. A series of analyses were conducted, including item drift to evaluate stability of the characteristics. The results suggest that empirical characteristics of the test forms and individual questions have remained relatively stable since the release policy was implemented. Specifically, statistical properties of the test forms have continued to perform similarly to test forms that were constructed prior to the release. Although the results of this study were promising, the context of this specific testing program may have offered additional protections such as a limited number of administrations that others may not. Therefore, testing/examining agencies and university programs may want to consider this strategy with appropriate caution.

Assessment in pediatrics clerkships: impact of strategies to solve item-sharing problems

“…In this case, both equating approach and the entire assessment procedure will be compromised. This is a controversial topic, because some works did not find differences in reused items difficulty 21 or in the overall students’ performance in tests with reused items, 22 whereas others point to a benefit or prior exposure to items. 23 Still, few works have addressed this issue so far and even fewer have directly studied the influence of reusing questions in medical clerkships assessment.…”

A Framework for Policies and Practices to Improve Test Security Programs: Prevention, Detection, Investigation, and Resolution (PDIR)

“…Wollack and Case () address protecting test security in the context of maintaining fair, or comparable, testing conditions for all examinees. A study of a single licensure examination indicates no deleterious effects on item and test form characteristics of publicly releasing a large number of items (Buckendahl & Gerrow, ), a strategy often considered as a way of undermining motivation to reconstruct or otherwise compromise test item content.…”

Should Items and Answer Keys of Small-Scale Exams Be Published?

“…However, the possibility that some or all of the test items have been previously used and are easily accessible to students will result in measurements focusing only on memorization, even though these items are developed to measure high-level cognitive skills. This may also greatly affect the psychometric properties of the items as well as the reliability, measurement, decision validity, and equalization of test scores (Buckendahl, Gerrow, & Pros, 2016;Wollack, Sung, & Kang, 2006;Wood, 2009). As a result, exams may lose their meaning and significance.…”