Assessing the Efficacy of ChatGPT in Solving Questions Based on the Core Concepts in Physiology

Abstract

Background and objective ChatGPT is a large language model (LLM) generative artificial intelligence (AI) chatbot trained through deep learning to produce human-like language skills and analysis of simple problems across a wide variety of subject areas. However, in terms of facilitating the transfer of learning in medical education, a concern has arisen that while AI is adept at applying surface-level understanding, it does not have the necessary in-depth knowledge to act at an expert level, particularly in addressing the core concepts. In this study, we explored the efficacy of ChatGPT in solving various reasoning questions based on the five core concepts applied to different modules in the subject of physiology. Materials and methods In this study, a total of 82 reasoning-type questions from six modules applicable to the five core concepts were created by the subject experts. The questions were used to chat with the conversational AI tool and the responses generated at first instance were considered for scoring and analysis. To compare the scores among various modules and five core concepts separately, the Kruskal-Wallis test along with post hoc analysis were used. Results The overall mean score for the modules (60 questions) was 3.72 ±0.26 while the average score obtained for the core concepts (60 questions) was 3.68 ±0.30. Furthermore, statistically significant differences (p=0.05 for modules and p=0.024 for core concepts) were observed among various modules as well as core concepts. Conclusion The significant differences observed in the scores among various modules and core concepts highlight the varying execution of the same software tool, thereby necessitating the need for further evaluation of AI-enabled learning applications to enhance the transfer of learning among undergraduates.

Is ChatGPT’s Knowledge and Interpretative Ability Comparable to First Professional MBBS (Bachelor of Medicine, Bachelor of Surgery) Students of India in Taking a Medical Biochemistry Examination?

“…The AI chatbot obtained higher marks than the mean marks of the students in almost all types of question sets except in MCQs based on logical reasoning (Table 2, Table 3, and Figure 4). This result demonstrates the potential power of the chat generative pre in the subjects of physiology, biochemistry, pathology, and microbiology, respectively, explored the possible potential of the AI chatbot to be a successful tool for answering questions requiring higher-order thinking in these subjects [17][18][19][20]. The observations of our study are similar to these studies and establish the fact that AI chatbots can answer in a human-like way, which requires rational thinking.…”

How Well Did ChatGPT Perform in Answering Questions on Different Topics in Gross Anatomy?

“…Speaking about different medical subjects, ChatGPT demonstrates enhanced performance in physiology, head and neck surgery, and biochemistry, particularly when confronted with diverse question types beyond MCQs. A study underscores ChatGPT's effectiveness in addressing reasoning questions related to core physiology concepts, achieving an approximately 74% correct response rate [28]. In head and neck surgery, ChatGPT exhibited notable success with correct responses in closed-ended questions (84.7%) and accurate diagnoses in clinical scenarios (81.7%).…”

Evaluating AI performance in answering questions related to thoracic anatomy

“…Research by Banerjee A et al highlighted ChatGPT's effectiveness in tackling reasoning questions across diverse physiology modules, achieving an impressive 74% correctness. 23 In head and neck surgery, it responded correctly to 84.7% of closed-ended questions. It provided accurate diagnoses in 81.7% of clinical scenarios, with room for improvement in procedural details and bibliographic references.…”