Large Language Models (LLMs) vs. Specialist Doctors: A Comparative Study on Health Information in specific medical domains. (Preprint)

Abstract

BACKGROUND Although Large Language Models (LLMs) such as ChatGPT show promise in providing specialized information, their quality require further evaluation, especially considering that these models are trained on internet text and the quality of health-related information available online varies widely. OBJECTIVE The aim of this study was to evaluate the performance of ChatGPT in the context of patient education for individuals with chronic diseases, comparing it with that of industry experts to elucidate its strengths and limitations. METHODS This evaluation was conducted by analyzing the responses of ChatGPT and specialist doctors to questions posed by patients with Inflammatory Bowel Disease (IBD), comparing their performance in terms of subjective accuracy, empathy, completeness, and overall quality, as well as readability to support objective analysis. RESULTS In a series of 1578 binary choice assessments, ChatGPT was preferred in 48.4% (95% CI, 45.9%-50.9%) of instances. There were 12 instances where ChatGPT's responses were unanimously preferred by all evaluators, compared to 17 instances for specialist doctors. In terms of overall quality, there was no significant difference between the responses of ChatGPT (3.98; 95% CI, 3.93-4.02) and those of specialist doctors (3.95; 95% CI, 3.90-4.00) (t=0.95, p=0.34), both being considered "good". Although differences in accuracy (t=0.48, p=0.63) and empathy (t=2.19, p=0.03) lacked statistical significance, the completeness of textual output (t=9.27, p=0.00) was a distinct advantage of the Large Language Model (ChatGPT). In the sections of the question where patients and doctors responded together (Q223-Q242), ChatGPT demonstrated superior performance (p=0.006). Regarding readability, no statistical difference was found between the responses of specialist doctors (median: 7th grade, Q1: 4th grade; Q3: 8th grade) and those of ChatGPT (median: 7th grade, Q1: 7th grade; Q3: 8th grade) according to the Mann-Whitney U test (p=0.09). The overall quality of ChatGPT's output exhibited strong correlations with other sub-dimensions (with empathy: r=0.842; with accuracy: r=0.839; with completeness: r=0.795), and there was also a high correlation between the sub-dimensions of accuracy and completeness (r=0.762). CONCLUSIONS ChatGPT demonstrated more stable performance across various dimensions. Its output of health information content is more structurally sound, addressing the issue of variability in individual specialist doctors' information output. ChatGPT's performance highlights its potential as an auxiliary tool for health information, despite limitations such as AI hallucinations. It is recommended that patients be involved in the creation and evaluation of health information to enhance the quality and relevance of the information.