Evaluating the Quality and Usability of Artificial Intelligence–Generated Responses to Common Patient Questions in Foot and Ankle Surgery

Abstract

Background: Artificial intelligence (AI) platforms, such as ChatGPT, have become increasingly popular outlets for the consumption and distribution of health care–related advice. Because of a lack of regulation and oversight, the reliability of health care–related responses has become a topic of controversy in the medical community. To date, no study has explored the quality of AI-derived information as it relates to common foot and ankle pathologies. This study aims to assess the quality and educational benefit of ChatGPT responses to common foot and ankle–related questions. Methods: ChatGPT was asked a series of 5 questions, including “What is the optimal treatment for ankle arthritis?” “How should I decide on ankle arthroplasty versus ankle arthrodesis?” “Do I need surgery for Jones fracture?” “How can I prevent Charcot arthropathy?” and “Do I need to see a doctor for my ankle sprain?” Five responses (1 per each question) were included after applying the exclusion criteria. The content was graded using DISCERN (a well-validated informational analysis tool) and AIRM (a self-designed tool for exercise evaluation). Results: Health care professionals graded the ChatGPT-generated responses as bottom tier 4.5% of the time, middle tier 27.3% of the time, and top tier 68.2% of the time. Conclusion: Although ChatGPT and other related AI platforms have become a popular means for medical information distribution, the educational value of the AI-generated responses related to foot and ankle pathologies was variable. With 4.5% of responses receiving a bottom-tier rating, 27.3% of responses receiving a middle-tier rating, and 68.2% of responses receiving a top-tier rating, health care professionals should be aware of the high viewership of variable-quality content easily accessible on ChatGPT. Level of Evidence: Level III, cross sectional study.

A Use Case for Generative AI in Medical Education

“…Furthermore, LLMs with GAI capabilities, such as ChatGPT and Med-PaLM, have the potential to answer medically related questions [ 4 ] and—intriguingly for the medical education community—can pass the USMLE [ 5 ]. Such a notable feat by LLMs necessitates reevaluation of the future of medical training and practice while carefully considering the relevant ethical issues and current limitations of GAI, such as their susceptibility for generating misinformation through a process known as “hallucination.” As GAI and LLMs become increasingly integrated in daily practice, similar and iteratively improved methodologies represent a way for educators and learners alike to benefit considerably by better aligning flashcards from the comprehensive AnKing deck with in-house curricula in preparation for medical licensing examinations such as USMLE Step 1.…”

“…Even considering their present shortcomings, LLMs may provide a unique opportunity to significantly impact medical education in the intermediate term, especially given the propensity of contemporary medical students to supplement didactic learning with web-based learning resources [3]. Furthermore, LLMs with GAI capabilities, such as ChatGPT and Med-PaLM, have the potential to answer medically related questions [4] and-intriguingly for the medical education community-can pass the USMLE [5]. Such a notable feat by LLMs necessitates reevaluation of the future of medical training and practice while carefully considering the relevant ethical issues and current limitations of GAI, such as their susceptibility for generating misinformation through a process known as "hallucination."…”