Clinical Reasoning and Knowledge Assessment of Rheumatology Residents Compared to AI Models: A Pilot Study

Abstract

Background: The integration of artificial intelligence (AI) in medicine has progressed from rule-based systems to advanced models and is showing potential in clinical decision-making. In this study, the psychological impact of AI collaboration in clinical practice is assessed, highlighting its role as a support tool for medical residents. This study aimed to compare clinical decision-making approaches of junior rheumatology residents with both trained and untrained AI models in clinical reasoning, pre-diagnosis, first-line, and second-line management stages. Methods: Ten junior rheumatology residents and two GPT-4 models (trained and untrained) responded to 10 clinical cases, encompassing diagnostic and treatment challenges in inflammatory arthritis. The cases were evaluated using the Revised-IDEA (R-IDEA) scoring system and additional case management metrics. In addition to scoring clinical case performance, residents’ attitudes toward AI integration in clinical practice were assessed through a structured questionnaire, focusing on perceptions of AI’s potential after reviewing the trained GPT-4’s answers. Results: Trained GPT-4 outperformed residents across all stages, achieving significantly higher median R-IDEA scores and superior performance in pre-diagnosis, first-line, and second-line management phases. Residents expressed a positive attitude toward AI integration, with 60% favoring AI as a supportive tool in clinical practice, anticipating benefits in competence, fatigue, and burnout. Conclusions: Trained GPT-4 models outperform junior residents in clinical reasoning and management of rheumatology cases. Residents’ positive attitudes toward AI suggest its potential as a supportive tool to enhance confidence and reduce uncertainty in clinical practice. Trained GPT-4 may be used as a supplementary tool during the early years of residency.