¿Es capaz “ChatGPT” de aprobar el examen MIR de 2022? Implicaciones de la inteligencia artificial en la educación médica en España

Abstract

Artificial intelligence and natural language processing models have made an entrance into the field of medical education. Among them, the ChatGPT model has been used to try to solve different international medical exams. However, there is no literature which addresses this phenomenon in Europe or other Spanish-speaking countries. The present paper aims at evaluating the ability to answer questions of the ChatGPT model in the 2022 MIR, which grants access to the Spanish postgraduate training system. To this end, a cross-sectional descriptive analysis has been carried out in which all the questions of the 2022 MIR exam have been solved by this technology. ChatGPT was able to answer 51.4% of the questions correctly, which is approximately 69 net answers on said exam. According to estimates for this year, it would have obtained a 7688 position, which would be slightly below the population’s median, but would allow it to pass the cut-off score and choose a large number of specialties. These results are similar to those obtained in the existing literature, slightly worse to those obtained by this tool in the American USMLE exams. The development of AI is an opportunity for medical students and residents to learn, but it is also a risk in many ways. It is essential to train future specialists in the new reality of artificial intelligence so that they are able to use them and obtain benefits in a reasoned and safe manner. La inteligencia artificial y los modelos de procesamiento de lenguaje natural han irrumpido con fuerza en el ámbito de la educación médica. Entre ellos, el modelo ChatGPT ha sido utilizado para intentar resolver distintos exámenes de medicina a nivel internacional. Sin embargo, prácticamente no existe literatura en Europa ni países de habla hispana. El presente trabajo pretende evaluar la capacidad de responder preguntas del modelo ChatGPT en el examen MIR 2022. Para ello, se ha llevado a cabo un análisis transversal y descriptivo en el que se han introducido la totalidad de las preguntas del examen MIR 2022 en dicho modelo. ChatGPT ha sido capaz de responder de manera acertada un 51,4% de las preguntas, lo que supone aproximadamente 69 netas en el examen MIR. Según estimaciones para este año, obtendría un 7688, lo que estaría ligeramente por debajo de la mediana de la población presentada, pero que le permitiría pasar la nota de corte y escoger un gran número de especialidades. El resultado es similar a los obtenidos en la bibliografía previa, ligeramente por debajo de los resultados obtenidos por dicha herramienta en los exámenes americanos USMLE. Este tipo de modelos suponen una oportunidad para el aprendizaje de los estudiantes de medicina y los residentes, pero también supone un riesgo en muchos sentidos. Es fundamental formar a los futuros especialistas en la nueva realidad de la inteligencia artificial para que sean capaces de utilizarlas y obtener beneficios de manera razonada y segura.

Performance of ChatGPT on the Peruvian National Licensing Medical Examination: Cross-Sectional Study

“…Another study on the Neurosurgery Oral Board Preparation Question Bank showed that GPT-4 performed with an accuracy of 82.6%, while GPT-3.5 achieved an accuracy of 62.4% [20]. However, in our study, GPT-3.5 performed better on the NLME compared to previous studies where it failed examinations, including the USMLE and Spanish, Japanese, and Chinese NLMEs [2,[21][22][23]. This can be explained by our use of a prompt that resembles the "chain-of-thought prompting approach," in which ChatGPT decomposes multistep problems into smaller and manageable steps to enhance accuracy [24].…”

Below average ChatGPT performance in medical microbiology exam compared to university students

“…Previous studies have evaluated the efficacy of AI-based models in addressing MCQs across diverse healthcare disciplines, yielding varying outcomes (Newton and Xiromeriti, 2023). These assessments encompassed the utilization of ChatGPT in scenarios such as United States Medical Licensing Examination (USMLE) exam, as well as the performance within fields such as parasitology and ophthalmology, among others in various languages (e.g., Japanese, Chinese, German, Spanish) (Antaki et al, 2023;Carrasco et al, 2023;Friederichs et al, 2023;Huh, 2023;Kung et al, 2023;Takagi et al, 2023;Xiao et al, 2023). A recent scoping review examined the influence of the ChatGPT model on examination outcomes, including instances where ChatGPT demonstrated superior performance compared to students, albeit in a minority of cases (Newton and Xiromeriti, 2023).…”

Evaluating ChatGPT Efficacy in Navigating the Spanish Medical Residency Entrance Examination (MIR): A New Horizon for AI in Clinical Medicine

“…A previous Spanish study submitted the same MIR examination to GPT3.5 and obtained a lower score (54.8%) than ours [26]. One explanation of the difference could be the prompt used.…”