Post-validation item analysis to assess the validity and reliability of multiple-choice questions at a medical college with an innovative curriculum

Abstract

Background In medical education, the need to obtain reliable and valid assessments is critical for the learning process. This study implemented a post-validation item analysis to create a supply of valid questions for incorporation into the question bank. Methods A cross-sectional study was performed in the College of Medicine, University of Bisha, Saudi Arabia. The study was targeting 250 items and 750 distractors from 2017 to 2020. The post-validation item analysis was done to evaluate the quality of the items using test-scoring and reporting software. Data were analysed by SPSS Version 25. Quantitative variables were expressed as mean (SD), while qualitative variables were expressed as number and percentage. An independent t-test was done to reveal the association between the item analysis parameters. A value of p<0.05 was considered statistically significant. Results The mean difficulty index (DIF I), discrimination index (DI) and distractors efficacy (DE) were 73.8, 0.26 and 73.5%, respectively. Of 250 items, 38.8% had an acceptable DIF I (30%–70%) and 66.4% had ‘good to excellent’ DI (>0.2). Of 750 distractors, 33.6%, 37%, 20% and 9.2% had zero, one, two and three non-functional distractors, respectively. The mean Kuder–Richardson was 0.76. The DIF I was significantly associated with DE (p=0.048). The post-validation item analysis of this study showed that a considerable proportion of questions had acceptable parameters and were recommended for item banking. However, some questions needed to be rephrased and reassessed or discarded. Conclusion Three-option multiple-choice questions should be considered for future examinations to improve the assessment process.

Item Analysis of Final Examination Questions for Social Studies in Junior High Schools through the ITEMAN Program

“…Even so, there are those who argue that the questions that are considered good are moderate questions, that is, questions whose index difficulty ranges from 0.26 to 0.75 (Quaigrain & Arhin, 2017). The various criteria have the tendency to say that the items with a difficulty index of less than 0.25 and more than 0.75 should be avoided or not used because they are too difficult or too easy and are therefore less reflective as good measuring tools (Adiga et al, 2021;Ibrahim & Yahia, 2021). The questions that are difficult need to be included in a test to motivate students who are smart, while the questions that are easy will raise the spirits of weak students.…”

Needs Assessment Report Adopting a Mixed-Method Design: Chinese Postgraduates’ Impressions of Critical Thinking Approach in Research Method Course

“…In terms of the reliability and validity of the semi-structured interview, the researcher finalized the semi-structured interview questions for the current study using an analysis of content validation and item objective congruence (IOC). IOC scores were calculated based on the equation (Yahia, 2022;Gavlvin, 2017) The IOC points are divided into three grading scales based on the items' consistency and conformity. All experts as well as the external raters were required to select one of the following three responses as the issued grade: +1 = Congruent with clear understanding; 0 = Uncertain or not sure whether item related to the study; -1 = Not understand or not congruent or related to this study.…”

Item analysis: the impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items

“…The validity and reliability of MCQs can be ensured pre-construction by the presence of the content material from which the MCQs will be constructed and a blueprint [ 7 – 10 ]. Item analysis is a post-examination method for ensuring the validity and reliability of MCQs [ 11 ]. It provides feedback to tutors about the constructed items and the coverage of the content materials from which items were created [ 12 – 14 ].…”