The Application of ChatGPT in Medicine: A Scoping Review and Bibliometric Analysis

Wu, Jie; Ma, Yingzhuo; Wang, Jun; Xiao, Mingzhao

doi:10.2147/jmdh.s463128

Cited by 7 publications

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

Moglia,

Georgiou,

Cerveri

et al. 2024

Artif Intell Rev

View full text Add to dashboard Cite

Large language models (LLMs) have the intrinsic potential to acquire medical knowledge. Several studies assessing LLMs on medical examinations have been published. However, there is no reported evidence on tests related to robot-assisted surgery. The aims of this study were to perform the first systematic review of LLMs on medical examinations and to establish whether ChatGPT, GPT-4, and Bard can pass the Fundamentals of Robotic Surgery (FRS) didactic test. A literature search was performed on PubMed, Web of Science, Scopus, and arXiv following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach. A total of 45 studies were analyzed. GPT-4 passed several national qualifying examinations with questions in English, Chinese, and Japanese using zero-shot and few-shot learning. Med-PaLM 2 obtained similar scores on the United States Medical Licensing Examination with more refined prompt engineering techniques. Five different 2023 releases of ChatGPT, one of GPT-4, and one of Bard were tested on FRS. Seven attempts were performed with each release. The pass score was 79.5%. ChatGPT achieved a mean score of 64.6%, 65.6%, 75.0%, 78.9%, and 72.7% respectively from the first to the fifth tested release on FRS vs 91.5% of GPT-4 and 79.5% of Bard. GPT-4 outperformed ChatGPT and Bard in all corresponding attempts with a statistically significant difference for ChatGPT (p < 0.001), but not Bard (p = 0.002). Our findings agree with other studies included in this systematic review. We highlighted the potential and challenges of LLMs to transform the education of healthcare professionals in the different stages of learning, by assisting teachers in the preparation of teaching contents, and trainees in the acquisition of knowledge, up to becoming an assessment framework of leaners.

show abstract

Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

Moglia,

Georgiou,

Cerveri

et al. 2024

Artif Intell Rev

View full text Add to dashboard Cite

show abstract

Assessing the Performance of ChatGPT in Answering Questions Regarding Congenital Bicuspid Aortic Valve

Barua

2024

Preprint

View full text Add to dashboard Cite

Background: Artificial Intelligence (AI) models, such as ChatGPT, are widely being used in academia as well as by the common public. In the field of medicine, the information obtained by the professionals as well as by the patients has significant advantages while posing valid concerns regarding healthcare delivery and utilization. Methods: To demonstrate the immense utility as well as potential concerns of using ChatGPT to gather medical information, a set of questions were posed to the chatbot regarding a hypothetical patient with congenital bicuspid aortic valve (BAV), and the answers were recorded and reviewed for their authenticity and utility. Results: While the ChatGPT provided detailed information about clinical pictures, treatment, and outcomes regarding BAV, the authenticity of the information could not be verified due to a lack of citations. Further, human aspects that would normally emerge in a nuanced doctor-patient communication were missing in the ChatGPT output. Conclusions: Although the performance of AI in medical care is expected to grow, imperfections and ethical concerns will remain a huge challenge in utilizing information from the chatbots alone without adequate communications with health providers despite having numerous advantages of this technology to society in many walks of human life.

show abstract

Assessing the Performance of ChatGPT in Answering Questions Regarding Congenital Bicuspid Aortic Valve

Barua

2024

Preprint

View full text Add to dashboard Cite

IMPORTANCE: Artificial Intelligence (AI) models, such as ChatGPT, are widely being used in academia as well as by the common public. In the field of medicine, the information obtained by the professionals as well as by the patients from the AI tools has significant advantages while at the same time posing valid concerns regarding the validity and adequacy of information regarding healthcare delivery and utilization. Therefore, it is important to vet these AI tools through the prism of practicing physicians. OBJECTIVE AND METHODS: To demonstrate the immense utility as well as potential concerns of using ChatGPT to gather medical information, a set of questions were posed to the chatbot regarding a hypothetical patient with congenital bicuspid aortic valve (BAV), and the answers were recorded and reviewed for their authenticity and utility. RESULTS: While the ChatGPT provided detailed information about clinical pictures, treatment, and outcomes regarding BAV, the authenticity of the information could not be verified due to a lack of citations. Further, human aspects that would normally emerge in a nuanced doctor-patient communication were missing in the ChatGPT output. CONCLUSIONS AND RELEVANCE: Although the performance of AI in medical care is expected to grow, imperfections and ethical concerns will remain a huge challenge in utilizing information from the chatbots alone without adequate communications with health providers despite having numerous advantages of this technology to society in many walks of human life.

show abstract

The Application of ChatGPT in Medicine: A Scoping Review and Bibliometric Analysis

Cited by 7 publications

References 38 publications

Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

Large language models in healthcare: from a systematic review on medical examinations to a comparative analysis on fundamentals of robotic surgery online test

Assessing the Performance of ChatGPT in Answering Questions Regarding Congenital Bicuspid Aortic Valve

Assessing the Performance of ChatGPT in Answering Questions Regarding Congenital Bicuspid Aortic Valve

Contact Info

Product

Resources

About