Evaluation of online chat-based artificial intelligence responses about inflammatory bowel disease and diet

Background and AimsGenerative Pre-trained Transformer-4 (GPT-4) is a large language model (LLM) trained on a variety of topics, including the medical literature. Nutrition plays a critical role in managing inflammatory bowel disease (IBD), with an unmet need for nutrition-related patient education resources. The aim of this study is to examine the accuracy and reproducibility of responses by GPT-4 to patient nutrition questions related to IBD.MethodsQuestions were curated from adult IBD clinic visits, Facebook, and Reddit. Two IBD-focused registered dieticians independently graded the accuracy and reproducibility of GPT-4’s responses while a third senior IBD-focused registered dietitian arbitrated. To ascertain reproducibility, each question was inputted twice into the model. Descriptive analysis is presented as counts and proportions.ResultsIn total, 88 questions were included. The model provided correct responses to 73/88 questions (83.0%), with 61 (69.0%) graded as comprehensive. A total of 15/88 (17%) responses were graded as mixed with correct and incorrect/outdated data. When examined by category, the model provided comprehensive responses to 10 (62.5%) questions related to “Nutrition and diet needs for surgery”, 12 (92.3%) “Tube feeding and parenteral nutrition”, 11 (64.7%) “General diet questions”, 10 (50%) “Diet for reducing symptoms/inflammation” and 18 (81.8%) “Micronutrients/supplementation needs”. The model provided reproducible responses to 81/88 (92.0%) questions.ConclusionGPT-4 provided comprehensive responses to the majority of questions, demonstrating the promising potential of LLMs as supplementary tools for IBD patients seeking nutrition-related information. However, 17% of responses contained incorrect information, highlighting the need for continuous refinement and validation of LLMs prior to incorporation into clinical practice. Future studies should focus on leveraging LLMs to enhance patient outcomes. Furthermore, efforts promoting patient and healthcare professional proficiency in using LLMs are essential to maximize their efficacy and facilitate personalized care.

show abstract

The Application of Large Language Models in Gastroenterology: A Review of the Literature

Maida,

Celsa,

Lau

et al. 2024

Cancers

View full text Add to dashboard Cite

Large language models (LLMs) are transforming the medical landscape by enhancing access to information, diagnostics, treatment customization, and medical education, especially in areas like Gastroenterology. LLMs utilize extensive medical data to improve decision-making, leading to better patient outcomes and personalized medicine. These models are instrumental in interpreting medical literature and synthesizing patient data, facilitating real-time knowledge for physicians and supporting educational pursuits in medicine. Despite their potential, the complete integration of LLMs in real-life remains ongoing, particularly requiring further study and regulation. This review highlights the existing evidence supporting LLMs’ use in Gastroenterology, addressing both their potential and limitations. Recent studies demonstrate LLMs’ ability to answer questions from physicians and patients accurately. Specific applications in this field, such as colonoscopy, screening for colorectal cancer, and hepatobiliary and inflammatory bowel diseases, underscore LLMs’ promise in improving the communication and understanding of complex medical scenarios. Moreover, the review discusses LLMs’ efficacy in clinical contexts, providing guideline-based recommendations and supporting decision-making processes. Despite these advancements, challenges such as data completeness, reference suitability, variability in response accuracy, dependency on input phrasing, and a lack of patient-generated questions underscore limitations in reproducibility and generalizability. The effective integration of LLMs into medical practice demands refinement tailored to specific medical contexts and guidelines. Overall, while LLMs hold significant potential in transforming medical practice, ongoing development and contextual training are essential to fully realize their benefits.

show abstract

Evaluation of online chat-based artificial intelligence responses about inflammatory bowel disease and diet

Cited by 3 publications

References 28 publications

Generative AI in Pediatric Gastroenterology

Generative AI in Pediatric Gastroenterology

Examining the Accuracy and Reproducibility of Responses to Nutrition Questions Related to Inflammatory Bowel Disease by Generative Pre-trained Transformer-4 (GPT-4)

The Application of Large Language Models in Gastroenterology: A Review of the Literature

Contact Info

Product

Resources

About