Aim: The aim of this study was to find ways to improve reliability of cutoff scores that are typically used to make high-stake decisions in dental education by empirically comparing two different rating methods, Yes/No and Percentage methods. Material and Methods: The two rating methods are commonly used when the Angoff's method is applied to determine a cutoff score that divides the examinees into minimally competent group (pass) and incompetent group (fail). The expert panel data were collected using both methods from 11 to 13 panel members in two consecutive years, respectively; The data were analysed within the generalisability theory framework to quantify relative influences of each factor (eg panel, item, rating rounds) on the variability of cutoff scores, standard error of measurement and panel agreement. Results: The results suggest that (a) the two methods can make a substantial difference in overall success rates for college senior students, (b) item-related variance components are generally large and whilst rater-related variance components are small, (c) standard errors of measurement for the cutoff scores decreased from Cohort 1 to Cohort 2 as the number of items are increased and as the expert panel members are more trained and (d) the Percentage method yielded higher agreement amongst the panel in both years. The results provide practical guidelines for dental educators who make efforts to control the quality of final competency exams and cutoff scores with respect to standard setting practices and panel data analysis. Conclusion: It can be concluded that evaluations with Percentage method results in more reliable outcomes compared to those with Yes/No method when criterion-referenced assessment is applied to determine the cutoff scores of competency tests at schools.

Background The present study was conducted to examine the operation status of the Competency-Based Curriculum (CBC), which has become the main curriculum in the medical schools in Korea, and prepare valid evaluation criteria consented by experts to make improvements. The evaluation criteria were derived based on a model built by combining the Context, Input, Process and Product (CIPP) evaluation model and the Kirkpatrick evaluation model, which are representative educational evaluation models. Methods Firstly, literature survey was performed and a semi-structured interview was conducted with 5 experts to develop a draft of the evaluation criteria. To verify the validity of the draft of the developed evaluation criteria, two surveys based on the modified Delphi methodology were conducted with a panel consisting of 20 experts. Results Based on the literature survey and the expert interview, a draft of the evaluation criteria was derived, including 5 evaluation areas, 18 evaluation items and 58 evaluation indicators. Two Delphi surveys were conducted to validate the evaluation criteria. The evaluation criteria that showed a relatively low content validity ratio (CVR) were corrected and complemented by reflecting the experts’ opinions to finally derive 5 evaluation areas, 16 evaluation items and 51 evaluation indicators. Conclusions The significance of the present study is that an evaluation model and its evaluation criteria suitable for the curriculums of the medical schools in Korea were developed with the consensus of the experts. The preparation of valid evaluation criteria will open up the possibility of improving the evaluation in medical schools, contributing to the improvement of the educational quality and the continued quality improvement of medical education.