Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health

Abstract

Introduction:ChatGPT is an artificial intelligence platform available to patients seeking medical advice. Traditionally, urology patients consulted official provider-created materials, particularly the Urology Care FoundationÔ (UCF). Today, men increasingly go online due to the rising costs of health care and the stigma surrounding sexual health. Online health information is largely inaccessible to laypersons as it exceeds the recommended American sixth to eighth grade reading level. We conducted a comparative assessment of patient education materials generated by ChatGPT vs UCF regarding men's health conditions.Methods: All 6 UCF men's health resources were identified. ChatGPT responses were generated using patient questions obtained from UCF. Adjusted ChatGPT responses were generated by prompting, "Explain it to me like I am in sixth grade." Textual analysis was performed using sentence, word, syllable, and complex word count. Six validated formulae were used for readability analysis. Two physicians independently scored responses for accuracy, comprehensiveness, and understandability. Statistical analysis involved Wilcoxon matched-pairs test.Results: ChatGPT responses were longer and more complex. Both UCF and ChatGPT failed official readability standards, although ChatGPT performed significantly worse across all 6 topics (all P < .001). Conversely, adjusted ChatGPT readability typically surpassed UCF, even meeting the recommended level for 2 topics. Qualitatively, UCF and ChatGPT had comparable accuracy, although ChatGPT had better comprehensiveness and worse understandability.Conclusions: When comparing readability, ChatGPT-generated education is less accessible than provider-written content, although neither meets the recommended level. Our analysis indicates that specific artificial intelligence prompts can simplify educational materials to meet national standards and accommodate individual literacy.

Both Patients and Plastic Surgeons Prefer Artificial Intelligence–Generated Microsurgical Information

“…Notably, ChatGPT has the known ability to adjust reading levels of texts based on education level. 15 Further study of implementation of this AI tool into health care may consider whether responses generated by ChatGPT retain clarity, comprehensiveness, and accuracy when reading level of responses are requested to be appropriate for the general patient population. It should also be noted that responses from the two sources were of varying length, which may have contributed to complexity bias and affected scoring from participants.…”

“…A possible explanation for why patients indicated a preference for more complex material could be a welldocumented cognitive error known as "complexity bias," wherein people subconsciously demonstrate a preference for the complicated over the simple. 14 This logical fallacy may have prompted non-medical individuals to give greater credence to the more complex AI-generated material over the more readable ASRM material, although both were well above recommended readability levels. Notably, ChatGPT has the known ability to adjust reading levels of texts based on education level.…”

Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review

“…Current automated simplification methods scored poorly due to grammatical errors, repetition, and inconsistencies in the autogenerated documents [ 68 ]. Artificial intelligence–derived text simplification methods may overcome these barriers by matching a document’s reading level to the readers’ needs, as shown in a study where ChatGPT was able to modify answers to men’s health condition questions to accommodate lower reading levels [ 69 , 70 ]. However, popularly used AI tools, such as ChatGPT, need considerable evaluation to minimize inaccurate information delivery and improve comprehensibility.…”

“…However, popularly used AI tools, such as ChatGPT, need considerable evaluation to minimize inaccurate information delivery and improve comprehensibility. Current studies indicate that these tools lack citations for the information they provide and cannot differentiate between low-quality and high-quality information [ 70 , 71 ].…”

Assessing the Utility of ChatGPT in Simplifying Text Complexity of Patient Educational Materials

“…Another study published by the American Urological Association compared readability between patient education materials created by urologists and responses from ChatGPT version 3.5. ChatGPT had significantly poorer readability than provider-created articles across all topics that were tested, despite being prompted to provide responses at a sixth-grade reading level (all p<0.001) [ 25 ]. It is important to identify opportunities and limitations for chatbot use in patient care settings so we can maximize its impact.…”