Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary Care (Preprint)

Abstract

BACKGROUND Large language models exhibiting human-level performance in specialised tasks are emerging; examples include GPT3.5 which underlies the processing of ChatGPT. OBJECTIVE Here, we evaluated the strengths and weaknesses of ChatGPT in primary care, using the MRGCP Applied Knowledge Test (AKT) as a medium. METHODS AKT questions were sourced from an online question bank and two AKT practice papers. 674 unique AKT questions were inputted to ChatGPT, with the model’s answers recorded and compared to correct answers provided by the RCGP. Each question was inputted twice, in separate ChatGPT sessions, with answers on repeated trials compared to gauge consistency. Subject difficulty was gauged by referring to examiners’ reports over the last five years. Novel explanations from ChatGPT—defined as information provided which was not inputted within the question of multiple answer choices—were recorded. Performance was analysed with respect to subject, difficulty, question source, and novel explanations to explore ChatGPT’s strengths and weaknesses. RESULTS Average overall performance was 60.17%, below the mean passing mark in the last two years (70.42%). Accuracy differed between sources (p=0.035, 0.059). ChatGPT’s performance varied with subject category (p=0.021, 0.015), but variation did not correlate with difficulty (ρ=-0.241, -0.238; p=0.191, 0.197). The proclivity of ChatGPT to provide novel explanations did not affect accuracy (p=1.000, 0.233). CONCLUSIONS Large language models are approaching human expert-level performance, although further development is required to match qualified primary care physicians in the AKT. Validated high-performance models may serve as assistants or autonomous clinical tools to ameliorate the general practice workforce crisis.

Utility of artificial intelligence‐based large language models in ophthalmic care

“…However, the judgement between GPT‐4 and its human counterparts (experts) was not yet conclusive and therefore warrants further investigation. Results were similar among the American Academy of Ophthalmology's Basic and Clinical Science Course (46.0%–84.3%), 41,42,46,55 Ophthoquestions (42.7%–84%), 41,47,48 Fellow of The Royal College of Ophthalmologists (FRCOphth) examination questions (32%–88.4%) 51,57 and Statpearls (55.5%–73.2%) 49 . In comparison, a lower score was observed in Brazil board examination questions (41.5%) 44 and higher in European board examinations (91%) 50 .…”

“…LLMs regularly outperformed the threshold of ophthalmological specialist examinations 40–57 . However, the accuracy of LLMs was 66.9% (22.4%–91%) while the ophthalmology trainees scored 68.4% (33%–75.7%—Table 3).…”

Large language models in health care: Development, applications, and challenges

“…Although LLMs have shown impressive performance across a range of NLP tasks, their efficacy in specialized tasks is limited [22]. A lack of domain‐specific knowledge in general LLMs hinders their ability to interpret technical terms and produce accurate, reasoned answers.…”

Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology

“…In addition to general NLP tasks, foundation LLMs have shown promising results in generalizing to unseen tasks even in medical question-answering requiring scientific expert knowledge. 14 , 15 , 16 , 17 , 18 These tasks require LLMs to understand the medical context, recall, and interpret relevant medical information in order to formulate an answer. Reported performance in ophthalmology has been mixed, but there appears to be potential to apply LLMs in eye health care applications if important limitations can be addressed.…”

“…Reported performance in ophthalmology has been mixed, but there appears to be potential to apply LLMs in eye health care applications if important limitations can be addressed. 14 , 15 , 16 , 17 , 18 Various strategies have been described to develop foundation LLMs with enhanced performance in clinical tasks. These include building domain-specific LLMs by pretraining with curated medical text, fine-tuning foundation LLMs with domain-specific medical data, or using innovative prompting strategies.…”