Evaluation of ChatGPT for Pelvic Floor Surgery Counseling

Abstract

Importance Large language models are artificial intelligence applications that can comprehend and produce human-like text and language. ChatGPT is one such model. Recent advances have increased interest in the utility of large language models in medicine. Urogynecology counseling is complex and time-consuming. Therefore, we evaluated ChatGPT as a potential adjunct for patient counseling. Objective Our primary objective was to compare the accuracy and completeness of ChatGPT responses to information in standard patient counseling leaflets regarding common urogynecological procedures. Study Design Seven urogynecologists compared the accuracy and completeness of ChatGPT responses to standard patient leaflets using 5-point Likert scales with a score of 3 being “equally accurate” and “equally complete,” and a score of 5 being “much more accurate” and much more complete, respectively. This was repeated 3 months later to evaluate the consistency of ChatGPT. Additional analysis of the understandability and actionability was completed by 2 authors using the Patient Education Materials Assessment Tool. Analysis was primarily descriptive. First and second ChatGPT queries were compared with the Wilcoxon signed rank test. Results The median (interquartile range) accuracy was 3 (2–3) and completeness 3 (2–4) for the first ChatGPT query and 3 (3–3) and 4 (3–4), respectively, for the second query. Accuracy and completeness were significantly higher in the second query (P < 0.01). Understandability and actionability of ChatGPT responses were lower than the standard leaflets. Conclusions ChatGPT is similarly accurate and complete when compared with standard patient information leaflets for common urogynecological procedures. Large language models may be a helpful adjunct to direct patient-provider counseling. Further research to determine the efficacy and patient satisfaction of ChatGPT for patient counseling is needed.

Letter to the Editor re: “Evaluation of ChatGPT for Pelvic Floor Surgery Counseling: Correspondence”

“…

We would like to comment on "Evaluation of ChatGPT for Pelvic Floor Surgery Counseling." 1 The purpose of the study was to evaluate ChatGPT responses in terms of completeness and accuracy in comparison to typical patient counseling pamphlets for urogynecological procedures. Using the Patient Education Materials Assessment Tool, urogynecologists evaluated the understandability and actionability of ChatGPT responses by assigning ratings on a Likert scale.

…”