Assessing ChatGPT’s capacity for clinical decision support in pediatrics: A comparative study with pediatricians using KIDMAP of Rasch analysis

Abstract

Background: The application of large language models in clinical decision support (CDS) is an area that warrants further investigation. ChatGPT, a prominent large language models developed by OpenAI, has shown promising performance across various domains. However, there is limited research evaluating its use specifically in pediatric clinical decision-making. This study aimed to assess ChatGPT’s potential as a CDS tool in pediatrics by evCDSaluating its performance on 8 common clinical symptom prompts. Study objectives were to answer the 2 research questions: the ChatGPT’s overall grade in a range from A (high) to E (low) compared to a normal sample and the difference in assessment of ChatGPT between 2 pediatricians. Methods: We compared ChatGPT’s responses to 8 items related to clinical symptoms commonly encountered by pediatricians. Two pediatricians independently assessed the answers provided by ChatGPT in an open-ended format. The scoring system ranged from 0 to 100, which was then transformed into 5 ordinal categories. We simulated 300 virtual students with a normal distribution to provide scores on items based on Rasch rating scale model and their difficulties in a range between −2 to 2.5 logits. Two visual presentations (Wright map and KIDMAP) were generated to answer the 2 research questions outlined in the objectives of the study. Results: The 2 pediatricians’ assessments indicated that ChatGPT’s overall performance corresponded to a grade of C in a range from A to E, with average scores of −0.89 logits and 0.90 logits (=log odds), respectively. The assessments revealed a significant difference in performance between the 2 pediatricians (P < .05), with scores of −0.89 (SE = 0.37) and 0.90 (SE = 0.41) in log odds units (logits in Rasch analysis). Conclusion: This study demonstrates the feasibility of utilizing ChatGPT as a CDS tool for patients presenting with common pediatric symptoms. The findings suggest that ChatGPT has the potential to enhance clinical workflow and aid in responsible clinical decision-making. Further exploration and refinement of ChatGPT’s capabilities in pediatric care can potentially contribute to improved healthcare outcomes and patient management.

Exploring Early Perceptions and Experiences of ChatGPT in Pediatric Critical Care: A Qualitative Study Among Healthcare Professionals

“…This aligns with the growing trend of AI integration in healthcare, where ease of use is paramount for widespread adoption [35]. However, the participants’ perception of ChatGPT as a “limited” source, primarily due to its dependence on pre-existing datasets and the absence of real-time and accurate data updates, highlights a critical aspect of AI in healthcare: the need for continuous AI models in learning and adaptation to verified yet evolving medical knowledge [36].…”

“…The application of ChatGPT as an organizer and information source reflects its perceived role as an adjunct, rather than a replacement, in clinical decision-making processes [36]. This is particularly noteworthy in the context of pediatric intensive care, where the complexity and variability of cases necessitate human expertise and judgment.…”

Exploring the role of large language models in radiation emergency response

“…In [39] it was found that ChatGPT correctly answered 74% of the trivia questions related to heart diseases. Specifically, the accuracy of ChatGPT scored impressively in the domains of coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%) [40] evaluated ChatGPT as a support tool for breast tumor board decision making [41] assessed ChatGPT's capacity for clinical decision support in paediatrics [7] evaluated the capacity of ChatGPT as a clinical decision support in triaging patients for appropriate imaging services [42] did a comparative analysis of humans and LLMs in decision making abilities. The analysis found that there was a moderate level of agreement between the decisions of humans and LLMs.…”

ChatGPT in urology practice: revolutionizing efficiency and patient care with generative artificial intelligence

“…Some authors performed studies in a clinical setting, assessing ChatGPT's performance in discriminating symptoms and providing possible diagnoses, hence assisting with the decision-making process. It appears that with the right inputs, ChatGPT is often able to perform well in the clinical scenario, addressing accurate diagnosis and possible treatment options [18]. Whiles et al [19] conducted the first study to examine the accuracy of ChatGPT in patient counselling responses.…”