Assessing ChatGPT Responses to Common Patient Questions Regarding Total Hip Arthroplasty

Abstract

Background: The contemporary patient has access to numerous resources on common orthopaedic procedures before ever presenting for a clinical evaluation. Recently, artificial intelligence (AI)-driven chatbots have become mainstream, allowing patients to engage with interfaces that supply convincing, human-like responses to prompts. ChatGPT (OpenAI), a recently developed AI-based chat technology, is one such application that has garnered rapid growth in popularity. Given the likelihood that patients may soon call on this technology for preoperative education, we sought to determine whether ChatGPT could appropriately answer frequently asked questions regarding total hip arthroplasty (THA). Methods: Ten frequently asked questions regarding total hip arthroplasty were posed to the chatbot during a conversation thread, with no follow-up questions or repetition. Each response was analyzed for accuracy with use of an evidence-based approach. Responses were rated as “excellent response not requiring clarification,” “satisfactory requiring minimal clarification,” “satisfactory requiring moderate clarification,” or “unsatisfactory requiring substantial clarification.” Results: Of the responses given by the chatbot, only 1 received an “unsatisfactory” rating; 2 did not require any correction, and the majority required either minimal (4 of 10) or moderate (3 of 10) clarification. Although several responses required nuanced clarification, the chatbot’s responses were generally unbiased and evidence-based, even for controversial topics. Conclusions: The chatbot effectively provided evidence-based responses to questions commonly asked by patients prior to THA. The chatbot presented information in a way that most patients would be able to understand. This resource may serve as a valuable clinical tool for patient education and understanding prior to orthopaedic consultation in the future.

Hip & Pelvis

“…Given its potential role in preoperative education, researchers from Nashville (Tennessee, USA) sought to assess the chatbot’s ability to provide accurate and appropriate answers to commonly asked questions about THA. 6 This should be an easy ‘win’ for an AI chatbot but, like everything, ChatGPT is only as good as the information it’s been trained with. The study was simple and involved posing ten frequently asked questions about THA to ChatGPT in a conversation thread without any follow-up questions.…”

Human versus artificial intelligence‐generated arthroplasty literature: A single‐blinded analysis of perceived communication, quality, and authorship source

“…Within medicine broadly—and orthopaedic surgery specifically—physicians rely on clear, accurate communication of clinical and basic science research to inform and evolve their practices. As access to powerful LLMs becomes increasingly universal, understanding whether such technologies can play a role in medical research and evidence‐based medicine is of growing importance 11,12 …”

“…As access to powerful LLMs becomes increasingly universal, understanding whether such technologies can play a role in medical research and evidence-based medicine is of growing importance. 11,12 This study's primary aim was to compare the confidence in authorship source of arthroplasty abstracts based on whether they were human-written or AI-generated as rated by blinded, fellowshiptrained arthroplasty surgeon reviewers. The secondary aims were to evaluate abstracts' rated effectiveness of communication and overall quality based on whether they were human-written or AI-generated.…”

Quality of ChatGPT Responses to Frequently Asked Questions in Carpal Tunnel Release Surgery

“…Previous literature within orthopedic surgery has been hesitantly optimistic regarding the accuracy and quality of responses given by ChatGPT. [3][4][5] However, to date, a few studies have evaluated ChatGPT's responses within the context of hand surgery. 5 In this study, we asked ChatGPT frequently asked patientfocused questions surgeons may receive in clinic from patients who are offered carpal tunnel release (CTR) surgery and evaluated the quality of its output.…”