Evaluating the Effectiveness of Artificial Intelligence–powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology

Abstract

Study Need and Importance: In 2022, Version 3.5 of ChatGPT, an artificial intelligenceepowered large language model (LLM) was released. Its adoption immediately burgeoned, and given that patients most commonly use the Internet as a primary medical information source, there is reason to believe they will adopt ChatGPT for medical information too. Urological patients may be particularly likely to use ChatGPT, as situations requiring urological care are broad ranging, with diverse treatment options from office procedures to major open surgery. No study has been done to assess ChatGPT's urological advice, and thus our study aimed to do so by assessing appropriateness, readability, and other qualities of ChatGPT-generated urological information. What We Found: The Figure contains a flowchart of the study methodology. Fourteen of 18 (77.8%) responses were deemed appropriate. No significant differences were found between treatment-and symptom-related questions, nor between oncologic, benign, and treatment-related questions. The most common reason from urologist-graders for low scores was missing information. The most concerning missing information includes a missed differential diagnosis of acute retention for a classic presentation of acute retention. The mean (SD) Flesch Reading Ease score was 35.5 (10.2), and the mean Flesh-Kincaid Reading Grade Level score was 13.5 (1.74), indicating college-level readability of responses.Limitations: We used a small sample of questions and physician-graders. ChatGPT responses to the same question change each time the question is asked. ChatGPT is only 1 artificial intelligenceepowered LLM of others that are available. Finally, ChatGPT is not designed specifically for medical use in mind. Interpretation for Patient Care: While the advent of LLMs is an exciting prospect for bridging the information gap between urology patients and timeconstrained physicians, they may give inappropriate advice, inappropriately triage emergent medical situations, and write at too high a readability level for the average patient. Patients should proceed to use with caution.

How artificial intelligence can provide information about subdural hematoma: Assessment of readability, reliability, and quality of ChatGPT, BARD, and perplexity responses

“…Researchers have particularly emphasized that natural language processors have limitations as a source of medical information. [ 42 ]…”

Artificial intelligence in the urology multidisciplinary team meeting: can ChatGPT suggest European Association of Urology guideline‐recommended prostate cancer treatments?

“…Much has been made of the practical utility of Chat Generative Pre‐Trained Transformer (ChatGPT) and other large language models (LLMs) in healthcare and urology. Although some papers have assessed their use as patient education tools [1,2], nothing has yet been published on their practical use within the cancer multidisciplinary team (MDT) meeting. We sought to test ChatGPT's treatment recommendations for patients with prostate cancer against those made in a real‐world MDT meeting.…”

This Month in Adult Urology

“…Davis et al (page 688) from Los Angeles, California, aimed to assess the suitability and readability of natural language processor–generated responses to urology-related inquiries. 2 Common patient questions were used as inputs in ChatGPT, covering oncologic, benign, and emergency categories, and assessed for accuracy, comprehensiveness, and clarity. The study found that 77.8% of the responses were deemed appropriate, with clarity receiving the highest scores.…”