Did faculty training on item construction and peer review improve the quality of MCQs? An observational study in a medical school

Abstract

Background: multiple choice questions (MCQs) test is the most popular assessment tool that used in medical schools, faculty training on construction of MCQs and peer review are the main factors that contributing to have a good quality MCQs test. This study aimed to evaluate the effect of these two factors on the quality of MCQs test. Methods: an extensive training program was conducted for the staff members who participated in teaching and assessment of undergraduate medical students of clinical biochemistry and psychiatric courses during the academic year 2020-2021. Post-test item analysis was used to compare pre-training with post-training item analysis parameters. The Chi-square test was applied, the level of significance was 5%. Results: The difficulty index showed great improvement in the number of items which lie in the acceptable range of difficulty from 16% and 25% before the training program, to 61% and 42% after the training program for psychiatry and clinical biochemistry courses respectively. This is significantly different (P<0.05). In psychiatry course the number of questions with negative discrimination power decreased from 36% before training program to 10% after it (P value >0.05). No significant improvement was noticed in biochemistry course regarding this aspect. The percentage of test items where all distracters were functioning was increased from 11% to 41% in psychiatry course (P value=0.0002), and from 12% to 46%, in biochemistry course (P value=0.0001). This is a significant improvement. Reliability after the training program and peer review processes also increased in both courses. However, this is not statistically significant. Conclusions: A dramatic improvement was observed in the quality of MCQs after peer review and training program. We recommend adopting training program on construction of MCQs to be mandatory for all medical school staff members in Saudi Arabia.