Quality of multiple-choice questions in medical internship qualification examination determined by item response theory at Debre Tabor University, Ethiopia

Belay, Lalem Menber; Yigzaw, Tegbar; Abebe, Fantu

doi:10.1186/s12909-022-03687-y

Cited by 11 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the actual di culty of the 2022 UGMLE was 60.9 ± 23.8. This result was consistent with the di culty values reported from similar studies conducted in Ethiopia, Pakistan, and India with comparable objectives (27)(28)(29). This may be because the raters have the same understanding of the minimum performance ability expected of borderline students on the UGMLE.…”

Section: Discussionsupporting

confidence: 90%

Experts’ prediction of the actual item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Gedamu,

Tadesse,

Moges

et al. 2024

Preprint

View full text Add to dashboard Cite

Background The ability of expert ratings to predict the difficulty level of items to identify test-taker performance is an important aspect of licensure exams. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of the raters involved in predicting item difficulty is central to setting credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual MCQ item difficulty of the 2022 Ethiopian undergraduate medicine licensure examination (UGMLE). Method 200 Multiple-Choice Questions (MCQs) of the 2022 cohort of 815 UGMLE examinees were used in this study and seven physicians participated in the difficulty level ratings. Then, analysis was conducted to understand experts' rating variation in predicting the computed actual difficulty levels of examinees. Descriptive statistics to profile the rater’s assessment results and ANOVA to examine differences between the raters' estimations of the difficulty levels of the MCQs were computed. Additionally, regression analysis was used to understand the interrater variations in their predictions of difficult levels compared to actual difficult levels of MCQs.It was also used to examine the proportion of variation that each rater explained in the predictions of actual difficulty levels across the whole set of MCQs and all UGMLE fourteen domains. Results The study revealed statistically significant differences in the mean difficult level ratings of some raters and moderate to high positive linear relationships with some exam domains. But also, statistically nonsignificant relationships were found in some other domains. Thus, considerations have been needed on the observed variability in difficult-level rating values across raters and UGMLE domains. In the regression results, experts' ratings accounted for 33% of the variations in the actual UGMLE difficulty levels. Between the expert-rated and actual difficulty levels, the regression model showed a moderately positive linear correlation (R = 0.57) that was statistically significant at p = .05. Conclusion This study demonstrated the complex nature of rating the level of difficulty of MCQs on UGMLE and the benefits of employing prior expert ratings. To improve a rater's rating accuracy in UGMLE, designing strategies in line with the changing nature of assessment methods guarantees to maintain the required reliability and validity of the exam.

show abstract

Section: Discussionsupporting

confidence: 90%

Experts’ prediction of the actual item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Gedamu,

Tadesse,

Moges

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The difficulty index was calculated as a quantitative assessment of the proportion of examinees answering each question correctly, estimating the individual question's difficulty level. The discrimination index refers to the question's capacity to differentiate between high and low performers on the overall test [19]. These two metrics were used to assess the validity of an assessment and to distinguish between examinees, thus enabling us to evaluate the performance of ChatGPT more accurately.…”

Section: Xsl • Fomentioning

confidence: 99%

Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study

Flores-Cohaila,

García-Vicente,

Vizcarra-Jiménez

et al. 2023

JMIR Med Educ

View full text Add to dashboard Cite

Background ChatGPT has shown impressive performance in national medical licensing examinations, such as the United States Medical Licensing Examination (USMLE), even passing it with expert-level performance. However, there is a lack of research on its performance in low-income countries’ national licensing medical examinations. In Peru, where almost one out of three examinees fails the national licensing medical examination, ChatGPT has the potential to enhance medical education. Objective We aimed to assess the accuracy of ChatGPT using GPT-3.5 and GPT-4 on the Peruvian National Licensing Medical Examination (Examen Nacional de Medicina [ENAM]). Additionally, we sought to identify factors associated with incorrect answers provided by ChatGPT. Methods We used the ENAM 2022 data set, which consisted of 180 multiple-choice questions, to evaluate the performance of ChatGPT. Various prompts were used, and accuracy was evaluated. The performance of ChatGPT was compared to that of a sample of 1025 examinees. Factors such as question type, Peruvian-specific knowledge, discrimination, difficulty, quality of questions, and subject were analyzed to determine their influence on incorrect answers. Questions that received incorrect answers underwent a three-step process involving different prompts to explore the potential impact of adding roles and context on ChatGPT’s accuracy. Results GPT-4 achieved an accuracy of 86% on the ENAM, followed by GPT-3.5 with 77%. The accuracy obtained by the 1025 examinees was 55%. There was a fair agreement (κ=0.38) between GPT-3.5 and GPT-4. Moderate-to-high-difficulty questions were associated with incorrect answers in the crude and adjusted model for GPT-3.5 (odds ratio [OR] 6.6, 95% CI 2.73-15.95) and GPT-4 (OR 33.23, 95% CI 4.3-257.12). After reinputting questions that received incorrect answers, GPT-3.5 went from 41 (100%) to 12 (29%) incorrect answers, and GPT-4 from 25 (100%) to 4 (16%). Conclusions Our study found that ChatGPT (GPT-3.5 and GPT-4) can achieve expert-level performance on the ENAM, outperforming most of our examinees. We found fair agreement between both GPT-3.5 and GPT-4. Incorrect answers were associated with the difficulty of questions, which may resemble human performance. Furthermore, by reinputting questions that initially received incorrect answers with different prompts containing additional roles and context, ChatGPT achieved improved accuracy.

show abstract

“…Introducing clinical scenarios into MCQs has improved the assessment of relevant knowledge and understanding and has helped to assess critical thinking, ethics, and professionalism [3,4]. The MCQs are assigned a cognitive level based on the level of Bloom's taxonomy of specific learning outcomes in the curriculum [3].…”

Section: Introductionmentioning

confidence: 99%

“…On the other hand, the 1 validity of the test is the ability of a test to measure accurately what it is supposed to measure. Evidence shows that validity is affected by many factors such as quality of test items, number of test items, qualification of item writing skills, and psychometric characteristics of items [4]. The language background of the students has been mentioned as an essential factor by some authors [5].…”

Section: Introductionmentioning

confidence: 99%

Assessment of the Quality of Multiple-Choice Questions in the Surgery Course for an Integrated Curriculum, University of Bisha College of Medicine, Saudi Arabia

Al Ameer

2023

Cureus

View full text Add to dashboard Cite

Introduction: Multiple-choice questions (MCQs) have been recognized as reliable assessment tools, and incorporating clinical scenarios in MCQ stems has enhanced their effectiveness in evaluating knowledge and understanding. Item analysis is used to assess the reliability and consistency of MCQs, indicating their suitability as an assessment tool. This study aims to ensure the competence of graduates in serving the community and establish an examination bank for the surgery course.

show abstract

Quality of multiple-choice questions in medical internship qualification examination determined by item response theory at Debre Tabor University, Ethiopia

Cited by 11 publications

References 39 publications

Experts’ prediction of the actual item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Experts’ prediction of the actual item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study

Assessment of the Quality of Multiple-Choice Questions in the Surgery Course for an Integrated Curriculum, University of Bisha College of Medicine, Saudi Arabia

Contact Info

Product

Resources

About