Opportunities for Repeat Testing: Practice Doesn't Always Make Perfect

Educational Measurement

Haist

2015

To mitigate security concerns and unfair score gains, credentialing programs routinely administer new test material to examinees retesting after an initial failing attempt. Counterintuitively, a small but growing body of recent research suggests that repeating the identical form does not create an unfair advantage. This study builds upon and extends this research by investigating changes in responses to specific items encountered on both the first and repeat attempts. Results indicate that scores gains for repeat examinees who were assigned an identical form were not different from repeat examinees who received a different, but parallel, form. Analyses of responses to individual items answered incorrectly on the initial attempt found that examinees 68% of the time selected the same incorrect option on their second attempt, suggesting repeaters are misinformed rather than uninformed. Implications for feedback, remediation, and retesting policies are discussed.

mentioning

confidence: 99%

Repeat Testing Effects on Credentialing Exams: Are Repeaters Misinformed or Uninformed?

Feinberg

Educational Measurement

Haist

2015

“…The cumulative findings indicate that score gains range from about one-fourth to three-fourths of a standard deviation (SD). The magnitude of the gain appears to vary with the ability level of the examinees, type of item format and test content, and whether examinees see an identical or alternate test form (Geving et al 2005;Hausknecht et al 2007;Lievens et al 2007;Raymond et al 2009). Although there has been very little research on retest effects on performance-based examinations, score gains appear to be larger than for multiplechoice tests (Raymond and Luciw-Dubas 2010;Schleicher et al 2010).…”

Section: Introductionmentioning

confidence: 92%

Measurement precision for repeat examinees on a standardized patient examination

Swygert

Kahraman

2011

Adv in Health Sci Educ

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement (SEM) for individual examinees. Conditional SEMs were computed for approximately 60,000 single-take examinees and 5,000 repeat examinees who completed the Step 2 Clinical Skills Examination(®) between 2007 and 2009. The study focused exclusively on ratings of communication and interpersonal skills. Conditional SEMs for single-take and repeat examinees were nearly indistinguishable across most of the score scale. US graduates and IMGs were measured with equal levels of precision at all score levels, as were examinees with differing levels of skill speaking English. There was no evidence that examinees with the largest score changes were measured poorly on either their first or second attempt. The large score increases for repeat examinees on this SP-based exam probably cannot be attributed to unexpectedly large errors of measurement.

“…Meta-analyses of cognitive ability tests used for personnel selection and college admissions report score gains averaging about one fourth of a standard deviation (SD) for examinees who receive a different test form on their second attempt and gains of one half of a standard deviation for examinees who receive the same test when they repeat (Hausknecht, Halpert, Di Paolo, Moriarty, & Gerrard, 2007;Kulik, Kulik, & Bangert, 1984). Score gains have also been reported for those who repeat written certification exams, except that there appears to be no additional advantage to receiving the same certification test on the second administration (Geving, Webb, & Davis, 2005;Raymond, Neustel, & Anderson, 2007;). Similar effects have been found for performance tests that use standardized patients.…”

mentioning

confidence: 88%

The Second Time Around: Accounting for Retest Effects on Oral Examinations

Luciw-Dubas

2010

Eval Health Prof

Years of research with high-stakes written tests indicates that although repeat examinees typically experience score gains between their first and subsequent attempts, their pass rates remain considerably lower than pass rates for first-time examinees. This outcome is consistent with expectations. Comparable studies of the performance of repeat examinees on oral examinations are lacking. The current research evaluated pass rates for more than 50,000 examinees on written and oral exams administered by six medical specialty boards for several recent years. Pass rates for first-time examinees were similar for both written and oral exams, averaging about 84% across all boards. Pass rates for repeat examinees on written exams were expectedly lower, ranging from 22% to 51%, with an average of 36%. However, pass rates for repeat examinees on oral exams were markedly higher than for written exams, ranging from 53% to 77%, with an average of 65%. Four explanations for the elevated repeat pass rates on oral exams are proposed, including an increase in examinee proficiency, construct-irrelevant variance, measurement error (score unreliability), and memorization of test content. Simulated data are used to demonstrate that roughly one third of the score increase can be explained by measurement error alone. The authors suggest that a substantial portion of the score increase can also likely be attributed to construct-irrelevant variance. Results are discussed in terms of their implications for making pass-fail decisions when retesting is allowed. The article concludes by identifying areas for future research.