Performance of ChatGPT on Nephrology Test Questions

Abstract

Background: ChatGPT is a novel tool that allows people to engage in conversations with an advanced machine learning model. ChatGPT's performance in the United States Medical Licensing Examination is comparable to a successful candidate’s performance. However, its performance in nephrology field remains undetermined. This study assessed ChatGPT's capabilities in answering nephrology test questions. Methods: Questions sourced from Nephrology Self-Assessment Program and Kidney Self-Assessment Program were used, each with multiple choice single answer questions. Questions containing visual elements were excluded. Each question bank was run twice using GPT-3.5 and GPT-4. Total accuracy rate, defined as the percentage of correct answers obtained by ChatGPT in either the first or second run, and the total concordance, defined as the percentage of identical answers provided by ChatGPT during both runs, regardless of their correctness, were used to assess its performance. Results: A comprehensive assessment was conducted on a set of 975 questions, comprising 508 questions from Nephrology Self-Assessment Program and 467 from Kidney Self-Assessment Program. GPT-3.5 resulted in a total accuracy rate of 51%. Notably, the employment of Nephrology Self-Assessment Program yielded a higher accuracy rate compared to Kidney Self-Assessment Program (58% vs. 44%; p<0.001). The total concordance rate across all questions was 78%, with correct answers exhibiting a higher concordance rate (84%) compared to incorrect answers (73%) (p<0.001). When examining various nephrology subfields, the total accuracy rates were relatively lower in electrolyte and acid-base disorder, glomerular disease, and kidney-related bone and stone disorders. The total accuracy rate of GPT-4’s response was 74%, higher than GPT-3.5 (p<0.001) but remained below the passing threshold and average scores of Nephrology examinees (77%). Conclusions: ChatGPT exhibited limitations regarding accuracy and repeatability when addressing nephrology-related questions. Variations in performance were evident across various subfields.

Artificial Intelligence in Heart Failure and Acute Kidney Injury: Emerging Concepts and Controversial Dimensions

“…This niche area calls for innovative tools and strategies. Emerging technologies like LLMs and chatbots [38][39][40][41][42][43] offer promising avenues to enhance patient care and are increasingly relevant for healthcare professionals in this field (Table 3).…”

Enhancing clinical decision‐making: Optimizing ChatGPT's performance in hypertension care

“…18 , 19 , 20 , 21 , 22 Our findings show that GPT‐4′s accuracy in responding to questions related to hypertension has increased by 20%, achieving a 77% success rate. 10 Despite the need for a subscription to access GPT‐4, its enhanced performance strongly supports its recommended use.…”

“… 6 AI, particularly through ChatGPT, introduces a new dimension to clinical decision‐making. 7 Studies have demonstrated ChatGPT's efficacy in providing relevant advice for cardiovascular disease prevention and hypertension management, 8 , 9 , 10 indicating its potential to improve clinical outcomes.…”

Evaluating ChatGPT's efficacy in assessing the safety of non-prescription medications and supplements in patients with kidney disease

“…11 ChatGPT-3.5 and ChatGPT-4 have garnered attention due to its ability to comprehend and respond to intricate queries, making it a prospective resource for individuals seeking drug-related information. 12 , 13 However, it is uncertain the extent to which these large language models can be used for guiding non-prescription medication and supplement safety considerations in patients with kidney disease. 14 We therefore sought to evaluate ChatGPT-3.5 and ChatGPT-4's performance in assessing the safety of over-the-counter medications and supplements in patients with kidney disease, while simultaneously comparing its performance to that of Micromedex.…”