Comparison of large language models in management advice for melanoma: Google's AI BARD, BingAI and ChatGPT

Abstract

Large language models (LLMs) are emerging artificial intelligence (AI) technology refining research and healthcare. Their use in medicine has seen numerous recent applications. One area where LLMs have shown particular promise is in the provision of medical information and guidance to practitioners. This study aims to assess three prominent LLMs—Google's AI BARD, BingAI and ChatGPT‐4 in providing management advice for melanoma by comparing their responses to current clinical guidelines and existing literature. Five questions on melanoma pathology were prompted to three LLMs. A panel of three experienced Board‐certified plastic surgeons evaluated the responses for reliability using reliability matrix (Flesch Reading Ease Score, the Flesch‐Kincaid Grade Level and the Coleman‐Liau Index), suitability (modified DISCERN score) and comparing them to existing guidelines. t‐Test was performed to calculate differences in mean readability and reliability scores between LLMs and p value <0.05 was considered statistically significant. The mean readability scores across three LLMs were same. ChatGPT exhibited superiority with a Flesch Reading Ease Score of 35.42 (±21.02), Flesch–Kincaid Grade Level of 11.98 (±4.49) and Coleman–Liau Index of 12.00 (±5.10), however all of these were insignificant (p > 0.05). Suitability‐wise using DISCERN score, ChatGPT 58 (±6.44) significantly (p = 0.04) outperformed BARD 36.2 (±34.06) and was insignificant to BingAI's 49.8 (±22.28). This study demonstrates that ChatGPT marginally outperforms BARD and BingAI in providing reliable, evidence‐based clinical advice, but they still face limitations in depth and specificity. Future research should improve LLM performance by integrating specialized databases and expert knowledge to support patient‐centred care.

Assessing the Responses of Large Language Models (ChatGPT-4, Gemini, and Microsoft Copilot) to Frequently Asked Questions in Breast Imaging: A Study on Readability and Accuracy

“…Hillmann et al [15] posed questions related to atrial fibrillation and cardiac implantable electronic devices to various chatbots, and similarly to our study, found that ChatGPT scored lower in terms of readability. However, Mu et al [16] and Seth et al [17] asked questions related to melanoma and rhinoplasty, respectively, to chatbots, and while the first study found no significant difference in readability among the chatbots, the other study conducted by Seth et al found ChatGPT and BARD (now called Gemini) to be superior in terms of readability. Haver et al [18], using the ChatGPT-3.5 version, requested to simplify the answers given to questions about breast cancer prevention and screening by entering an additional prompt into ChatGPT, and found that ChatGPT's responses were statistically significantly simplified compared to the original ones.…”

Can AI Answer My Questions? Utilizing Artificial Intelligence in the Perioperative Assessment for Abdominoplasty Patients

“…Recommendations by the American Medical Association and the National Institutes of Health suggest that materials related to plastic surgery should be written at a sixth-to eighthgrade reading level [47]. However, recent research indicates that the readability of LLMs surpasses these recommendations, requiring a higher level of patient understanding [48,49]. This discrepancy could potentially undermine the relationship between patients and healthcare providers, representing a significant barrier to the effective implementation of AI-driven chatbot perioperative tools in plastic surgery contexts.…”

The Comparative Diagnostic Capability of Large Language Models in Otolaryngology

“…Related research has utilized LLM's semiotic capacity to demonstrate its significant aptitude in answering patient questions and providing management/treatment advice for disease and symptomologies in otolaryngology and across specialties. [32][33][34][35][36] The potential of these models does not stop there; this study's demonstration of LLM language abilities with simple medical information could be used to streamline documentation and triage in the ER or even assist in translation and readability services in the clinic. While this paper's findings are specific to otolaryngology, a broader perspective must be emphasized due to the vast capabilities of LLMs.…”