Evaluation of Prompts to Simplify Cardiovascular Disease Information Using a Large Language Model

Abstract

AI chatbots powered by large language models (LLMs) are emerging as an important source of public-facing medical information. Generative models hold promise for producing tailored guidance at scale, which could advance health literacy and mitigate well-known disparities in the accessibility of health-protective information. In this study, we highlight an important limitation of basic approaches to AI-powered text simplification: when given a zero-shot or one-shot simplification prompt, GPT-4 often responds by omitting critical details. To address this limitation, we developed a new prompting strategy, which we term rubric prompting. Rubric prompts involve a combination of a zero-shot simplification prompt with brief reminders about important topics to address. Using rubric prompts, we generate recommendations about cardiovascular disease prevention that are more complete, more readable, and have lower syntactic complexity than baseline responses produced without prompt engineering. This analysis provides a blueprint for rigorous evaluation of AI model outputs in medicine.