A Short Note on the Relationship Between Pass Rate and Multiple Attempts

Abstract

For a certification, licensure, or placement exam, allowing examinees to take multiple attempts at the test could effectively change the pass rate. Change in the pass rate can occur without any change in the underlying latent trait, and can be an artifact of multiple attempts and imperfect reliability of the test. By deriving formulae to compute the pass rate under two definitions, this article provides tools for testing practitioners to compute and evaluate the change in the expected pass rate when a certain (maximum) number of attempts are allowed without any change in the latent trait. This article also includes a simulation study that considers change in ability and differential motivation of examinees to retake the test. Results indicate that the general trend shown by the analytical results is maintained-that is, the marginal expected pass rate increases with more attempts when the testing volume is defined as the total number of test takers, and decreases with more attempts when the testing volume is defined as the total number of test attempts.

Revisiting Retake Policy: Analyzing the Success Rates of Examinees With Multiple Attempts on the United States Medical Licensing Examination

“…In a similar fashion, numerous studies have shown false-positive decisions increase along with the number of repeat attempts, essentially lowering the cut score for repeaters. [8][9][10][11] Thus, retake policies must consider not only examinee fairness as seen through the lens of false-negative "failers" but also the public protection purpose of the program as seen through the lens of false-positive "passers. " 12 The United States Medical Licensing Examination (USMLE) is one example of a licensure program whose retest policies seek to balance the public protection, individual access, equity, and wellness issues that underlie false positives and false negatives.…”

What Do We Mean When We Say “Pass Rate”?

“…A study by Cheng and Liu () provides some insight into how pass rate definition may impact results. Their research derived analytical formulas and used a simulation to illustrate how the total attempts and terminal pass rates changed with the number of attempts allowed at an exam.…”

“…One can also infer from their work that if only a single attempt is allowed that the terminal and total attempts pass rates would be equal. However, Cheng and Liu () do not consider the first attempt or repeat attempts pass rates in their study, or state how these pass rates may relate to the total attempts or terminal attempts pass rates. They also do not provide an empirical example showing how the pass rates compare to each other using data from an actual testing program.…”

“…As Clauser and Wainer (2016) point out, a common policy in credentialing programs is to allow examinees to retake part or the entire exam if they do not pass on their first attempt. The policy on exam retakes differs across organizations (Cheng & Liu, 2016;Clauser & Wainer, 2016;Grabovsky & Wainer, 2017a, 2017b. Some organizations allow a specific number of retakes over a particular time period, while other organizations may allow unlimited retakes.…”

“…There are generally three ways of defining the pass rate (see Cheng & Liu, ). The first approach is to define the pass rate as the number of test takers who pass over the number of test takers who took an attempt at the exam (i.e., the terminal pass rate ).…”