Comparing AI-Generated and Clinician-Created Personalized Self-Management Guidance for Knee Osteoarthritis Patients: A Blinded Observational Study (Preprint)

Abstract

BACKGROUND Background: Personalized education is crucial for effective knee osteoarthritis (OA) management, but providing it remains challenging due to imbalanced patient-provider ratio and limited resources. This study explores the potential of GPT-4, a large language model, in generating tailored self-management guidance and compares its performance with physician-provided advice. OBJECTIVE Objective：This study aims to evaluate the effectiveness of GPT-4 in generating personalized education materials for patients with knee osteoarthritis (OA) and compare it with experienced clinicians. Specifically, the comparison is made in terms of efficiency, readability, accuracy, personalization, comprehensiveness, and safety. By leveraging patient data from previous trials, it is evaluated whether AI can improve the quality and accuracy of patient education and evaluate its potential in improving patient care and outcomes. METHODS Methods: A two-phase, blinded, observational study was conducted using patient data from a previous trial. In phase one, two experienced orthopedic specialists created personalized education materials. In phase two, the same data were input into GPT-4 by a physician to generate content. Materials were evaluated for efficiency (words per minute), readability (Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, and SMOG Index), accuracy, personality, comprehensiveness, and safety. RESULTS Results: GPT-4 demonstrated higher efficiency than clinicians (median 530.03 vs. 37.29 WPM, P < 0.001). GPT-4 content exhibited superior readability on the Flesch-Kincaid grade level, Gunning Fog Index, and SMOG Index (P < 0.001). Expert evaluations revealed that GPT-4 outperformed clinicians in accuracy (5.307 ± 1.731 vs. 4.76 ± 1.098, P = 0.047), personality (54.32 ± 6.212 vs. 33.2 ± 5.395, P < 0.001), comprehensiveness (51.74 ± 6.471 vs. 35.26 ± 6.657, P < 0.001), and safety (median 61 vs. 50, P < 0.001). CONCLUSIONS Conclusion: GPT-4 shows promise in generating high-quality, personalized patient education for knee OA, surpassing human experts. This study provides novel evidence for the potential of AI in enabling precise and intelligent patient education. Further research is needed to validate the findings in larger populations and assess the impact on patient outcomes.