ChatGPT Performance in the UK Medical Licensing Assessment: How to Train the Next Generation?

Al-Shakarchi, Nader; Haq, Ikram U.

doi:10.1016/j.mcpdig.2023.06.004

Cited by 3 publications

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Currently, there are no known studies on the consistency of answers generated by ChatGPT from a medical education context. Notably, Al-Shakarachi and Haq ( 22 ) also conducted a study on ChatGPT performance in UKMLA practice papers and found it achieved 100% accuracy in emergency medicine, palliative care, and otolaryngology in UKMLA practice papers. Our study has found ChatGPT performed differently in these specialties with 8/15 (53.3%), 3/5 (60.0%), and 2/3 (66.8%) SBAs answered correctly, respectively.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment

Lai,

Wu,

Hsu

et al. 2023

Front. Med.

View full text Add to dashboard Cite

IntroductionRecent developments in artificial intelligence large language models (LLMs), such as ChatGPT, have allowed for the understanding and generation of human-like text. Studies have found LLMs abilities to perform well in various examinations including law, business and medicine. This study aims to evaluate the performance of ChatGPT in the United Kingdom Medical Licensing Assessment (UKMLA).MethodsTwo publicly available UKMLA papers consisting of 200 single-best-answer (SBA) questions were screened. Nine SBAs were omitted as they contained images that were not suitable for input. Each question was assigned a specialty based on the UKMLA content map published by the General Medical Council. A total of 191 SBAs were inputted in ChatGPT-4 through three attempts over the course of 3 weeks (once per week).ResultsChatGPT scored 74.9% (143/191), 78.0% (149/191) and 75.6% (145/191) on three attempts, respectively. The average of all three attempts was 76.3% (437/573) with a 95% confidence interval of (74.46% and 78.08%). ChatGPT answered 129 SBAs correctly and 32 SBAs incorrectly on all three attempts. On three attempts, ChatGPT performed well in mental health (8/9 SBAs), cancer (11/14 SBAs) and cardiovascular (10/13 SBAs). On three attempts, ChatGPT did not perform well in clinical haematology (3/7 SBAs), endocrine and metabolic (2/5 SBAs) and gastrointestinal including liver (3/10 SBAs). Regarding to response consistency, ChatGPT provided correct answers consistently in 67.5% (129/191) of SBAs but provided incorrect answers consistently in 12.6% (24/191) and inconsistent response in 19.9% (38/191) of SBAs, respectively.Discussion and conclusionThis study suggests ChatGPT performs well in the UKMLA. There may be a potential correlation between specialty performance. LLMs ability to correctly answer SBAs suggests that it could be utilised as a supplementary learning tool in medical education with appropriate medical educator supervision.

show abstract

Section: Discussionmentioning

confidence: 99%

“…The standard of the AKT is set by a national panel of experts from medical schools across the UK ( 21 ). Recently, a study reported that ChatGPT correctly answered 140/191 SBAs (73.3%) on the UKMLA ( 22 ).…”

Section: Introductionmentioning

confidence: 99%

Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment

Lai,

Wu,

Hsu

et al. 2023

Front. Med.

View full text Add to dashboard Cite

show abstract

ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review

Newton,

Xiromeriti

2024

Assessment & Evaluation in Higher Education

View full text Add to dashboard Cite

Evaluation of a Large Language Model on the American Academy of Pediatrics' PREP Emergency Medicine Question Bank

Ramgopal,

Varma,

Gorski

et al. 2024

Pediatr Emer Care

View full text Add to dashboard Cite

Background Large language models (LLMs), including ChatGPT (Chat Generative Pretrained Transformer), a popular, publicly available LLM, represent an important innovation in the application of artificial intelligence. These systems generate relevant content by identifying patterns in large text datasets based on user input across various topics. We sought to evaluate the performance of ChatGPT in practice test questions designed to assess knowledge competency for pediatric emergency medicine (PEM). Methods We evaluated the performance of ChatGPT for PEM board certification using a popular question bank used for board certification in PEM published between 2022 and 2024. Clinicians assessed performance of ChatGPT by inputting prompts and recording the software's responses, asking each question over 3 separate iterations. We calculated correct answer percentages (defined as correct in at least 2/3 iterations) and assessed for agreement between the iterations using Fleiss' κ. Results We included 215 questions over the 3 study years. ChatGPT responded correctly to 161 of PREP EM questions over 3 years (74.5%; 95% confidence interval, 68.5%–80.5%), which was similar within each study year (75.0%, 71.8%, and 77.8% for study years 2022, 2023, and 2024, respectively). Among correct responses, most were answered correctly on all 3 iterations (137/161, 85.1%). Performance varied by topic, with the highest scores in research and medical specialties and lower in procedures and toxicology. Fleiss' κ across the 3 iterations was 0.71, indicating substantial agreement. Conclusion ChatGPT provided correct answers to PEM responses in three-quarters of cases, over the recommended minimum of 65% provided by the question publisher for passing. Responses by ChatGPT included detailed explanations, suggesting potential use for medical education. We identified limitations in specific topics and image interpretation. These results demonstrate opportunities for LLMs to enhance both the education and clinical practice of PEM.

show abstract

ChatGPT Performance in the UK Medical Licensing Assessment: How to Train the Next Generation?

Cited by 3 publications

References 3 publications

Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment

Evaluating the performance of ChatGPT-4 on the United Kingdom Medical Licensing Assessment

ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review

Evaluation of a Large Language Model on the American Academy of Pediatrics' PREP Emergency Medicine Question Bank

Contact Info

Product

Resources

About