Efficacy of ChatGPT in solving attitude, ethics, and communication case scenario used for competency-based medical education in India: A case study

Abstract

BACKGROUND: Competency-based medical education (CBME) is a method of medical training that focuses on developing learners’ competencies rather than simply assessing their knowledge and skills. Attitude, ethics, and communication (AETCOM) are important components of CBME, and the use of artificial intelligence (AI) tools such as ChatGPT for CBME has not been studied. Hence, we aimed to assess the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India. MATERIALS AND METHODS: A total of 11 case scenarios were developed based on the AETCOM competencies. The scenarios were presented to ChatGPT, and the responses generated by ChatGPT were evaluated by three independent experts by awarding score ranging from 0 to 5. The scores were compared with a predefined score of 2.5 (50% accuracy) and 4 (80% accuracy) of a one-sample median test. Scores among the three raters were compared by the Kruskal–Wallis H test. The inter-rater reliability of the evaluations was assessed using the intraclass correlation coefficient (ICC). RESULTS: The mean score of solution provided by ChatGPT was 3.88 ± 0.47 (out of 5), indicating an accuracy of approximately 78%. The responses evaluated by three raters were similar (Kruskal–Wallis H P value 0.51), and the ICC value was 0.796, which indicates a relatively high level of agreement among the raters. CONCLUSION: ChatGPT shows moderate capability in solving AETCOM case scenarios used for CBME in India. The inter-rater reliability of the evaluations suggests that ChatGPT’s responses were consistent and reliable. Further studies are needed to explore the potential of ChatGPT and other AI tools in CBME and to determine the optimal use of these tools in medical education.

Exploring ChatGPT’s potential in the clinical stream of neurorehabilitation

“…Furthermore, recent studies conducted by Roy et al (2024) and Suri et al (2024) highlight the promising potential of ChatGPT in facilitating communication and delivering personalized support to neurorehabilitation patients. In line with these studies, Cheng et al (2023b) , Brown et al (2023) , and Mittal and Dhar (2023) have defined tangible examples of ChatGPT utilization and its impact on patient outcomes.…”

“…The use of ChatGPT in the medical field raises important ethical and practical questions ( Floridi et al, 2018 ). First, ChatGPT may lack the ability to suggest humanistic, non-pharmacological approaches ( Roy et al, 2024 ; Suri et al, 2024 ). This is attributed to its learning method, mainly based on pattern recognition and statistical associations ( Chen et al, 2024 ), as discussed in a study conducted by Suri et al (2024) .…”

“…This raises further ethical concerns regarding the security and integrity of medical information processed by AI, as noted by Roman et al (2023) . Therefore, the use of ChatGPT as a decision-making tool requires special caution and verification by qualified medical professionals ( Roy et al, 2024 ; Suri et al, 2024 ). Indeed, to ensure patient safety and the effectiveness of therapy, it is imperative to ensure that ChatGPT is used in conjunction with expert supervision ( Rodriguez et al, 2024 ).…”