How to improve reliability of cut‐off scores in dental competency exam: A comparison of rating methods in standard setting

Abstract

Aim: The aim of this study was to find ways to improve reliability of cutoff scores that are typically used to make high-stake decisions in dental education by empirically comparing two different rating methods, Yes/No and Percentage methods. Material and Methods: The two rating methods are commonly used when the Angoff's method is applied to determine a cutoff score that divides the examinees into minimally competent group (pass) and incompetent group (fail). The expert panel data were collected using both methods from 11 to 13 panel members in two consecutive years, respectively; The data were analysed within the generalisability theory framework to quantify relative influences of each factor (eg panel, item, rating rounds) on the variability of cutoff scores, standard error of measurement and panel agreement. Results: The results suggest that (a) the two methods can make a substantial difference in overall success rates for college senior students, (b) item-related variance components are generally large and whilst rater-related variance components are small, (c) standard errors of measurement for the cutoff scores decreased from Cohort 1 to Cohort 2 as the number of items are increased and as the expert panel members are more trained and (d) the Percentage method yielded higher agreement amongst the panel in both years. The results provide practical guidelines for dental educators who make efforts to control the quality of final competency exams and cutoff scores with respect to standard setting practices and panel data analysis. Conclusion: It can be concluded that evaluations with Percentage method results in more reliable outcomes compared to those with Yes/No method when criterion-referenced assessment is applied to determine the cutoff scores of competency tests at schools.

Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study

“…This study compared the probabilities of correct answers for a person with minimum competency between PA and YNA methods. Similar to previous studies [10,11], the reliability coefficients of PA were higher than those of YNA, and the cut scores of YNA were higher than those of PA, so the failure rate was also higher. Because the YNA method is faster and easier to decide for panelists, but the reliability is relatively lower, the YNA is good for use in local, medium-stakes performance exams [5].…”

“…According to the Hofstee results in Fig. 1 , the maximum acceptable cutoff score was 69.5%, the minimum acceptable cutoff score was 56.25%, and the final estimated cutoff score was 61.9% for the 2017 KMLE [ 10 ]. The cutoff scores with the PA, YNA with 60% (YNA-60), and with M-SD, were acceptable based on the Hofstee results.…”

“…According to the Hofstee results in Fig. 1, the maximum acceptable cut score was 69.5%, the minimum acceptable cut score was 56.25%, and the final estimated cut score was 61.9% for the 2017 KMLE [10].…”

“…Except for 4 droppers, the passing rate was 92.8% (3,105 passed and 158 failed). The mean difficulty was 72.1%, the mean discrimination was 0.17, the reliability was 0.926, and the standard deviation (SD) was 7.33 [ 10 ].…”

“…To find the cause of the error, an analysis based on generalizability theory is usually used. Generalizability theory [ 9 ] has been commonly applied to quantify the relative influences of each factor (e.g., panelists, items, rating rounds) on the variability (reliability coefficient) of cutoff scores, standard error (SE) of measurement, and panelist agreement [ 10 ].…”