ChatGPT-3.5 and ChatGPT-4 dermatological knowledge level based on the Specialty Certificate Examination in Dermatology

Abstract

Background The global use of artificial intelligence has the potential to revolutionize the healthcare industry. Despite the fact that artificial intelligence is becoming more popular, there is still a lack of evidence on its use in dermatology. Objective The study aimed to determine the capacity of ChatGPT-3.5 and ChatGPT-4 to support dermatological knowledge and clinical decision-making in medical practice. Methods Three dermatology specialty certificate tests, in English and Polish, consisting of 120 single-best-answer, multiple-choice format questions each, were used to assess ChatGPT-3.5 and ChatGPT-4 performance. Results ChatGPT-4 exceeded the 60% pass rate in every performed test, with a minimum of 80% and 70% correct answers for the English and Polish versions, respectively. ChatGPT-4 performed significantly better on each exam (p<0.01), regardless of the language, compared to ChatGPT-3.5. Furthermore, ChatGPT-4 answered clinical picture-type questions with an average accuracy of 92.98% and 84.21% for English and Polish questions respectively. The difference between the tests in Polish and English did not turn out to be significant but still, ChatGPT-3.5 and ChatGPT-4 in English performed better overall than in Polish by an average of 8 percentage points for each test. Incorrect ChatGPT answers were highly correlated with a lower difficulty index, which denotes questions with higher difficulty in most of the tests. (p<0.05) Conclusion The dermatological knowledge level of ChatGPT was high, with a significantly better performance of ChatGPT-4 than ChatGPT-3.5. Although the use of ChatGPT will not replace the doctor's final decision, physicians should support artificial intelligence development in dermatology to raise the standards of medical care.

Accuracy and Repeatability of ChatGPT Based on a Set of Multiple-Choice Questions on Objective Tests of Hearing

“…When our results are placed alongside studies from other disciplines that have examined ChatGPT's performance on single-answer and multiple-choice questions, we observe a similar trend -a roughly 20% performance improvement with ChatGPT 4 over ChatGPT 3.5 [6,[23][24][25]. Studies in otolaryngology, closely related to audiology, have also shown lower performance scores for ChatGPT 3.5, supporting our findings of improved outcomes with the newer version [26,27].…”

“…Presently, two versions of ChatGPT are accessible to the general public: the freely available ChatGPT 3.5, based on an earlier LLM, and the more advanced, subscription-based ChatGPT 4. Research has shown that ChatGPT 4 outperforms its predecessor, demonstrating superior accuracy in various fields such as dermatology, where it achieved 80-85% accuracy compared to 60-70% for ChatGPT 3.5 [6]. Similar improvements are observed in orthopedic assessments and in general medical examinations, with ChatGPT 4 consistently outperforming ChatGPT 3.5 [7,8].…”

“…Accuracy can vary significantly based on the topic and the complexity of the questions [28]. For instance, studies have shown that the accuracy of ChatGPT 3.5 can range from as high as 70% [6] to as low as 43% [27]. A critical question then arises: are the responses consistent across different trials at various times?…”

ChatGPT provides inconsistent risk-stratification of patients with atraumatic chest pain

“…While ChatGPT-4 demonstrates proficiency in conducting general conversations in multiple languages, its capacity for medical reasoning and understanding remains to be thoroughly assessed. Several studies have indicated ChatGPT’s competence in executing single medical task commands, such as answering multiple-choice questions from exams like the United States Medical Licensing Exam [ 5 , 6 ] and various medical specialty exams [ 6 ]. However, ChatGPT-4 struggles with logical questions [ 7 ] and occasionally fabricates responses [ 8 ].…”

ChatGPT Provides Inconsistent Risk-Stratification of Patients With Atraumatic Chest Pain

“…While ChatGPT can carry on general conversations in multiple languages, its ability to reason and understand the language of medicine needs further evaluation. Multiple studies show that ChatGPT does well on single medical task commands such as answering multiple-choice examination questions such as the United States Medical Licensing Exam (5,6) and medical specialty exams (6). However, ChatGPT needs help with logical questions (7) and tends to fabricate answers (8).…”