Accuracy and reproducibility of ChatGPT's free version answers about endometriosis

Abstract

ObjectiveTo evaluate the accuracy and reproducibility of ChatGPT's free version answers about endometriosis for the first time.MethodsDetailed internet searches to identify frequently asked questions (FAQs) about endometriosis have been performed. Scientific questions were prepared in accordance with the European Society of Human Reproduction and Embryology (ESHRE) endometriosis guidelines. An experienced gynecologist gave a score of 1–4 for each ChatGPT answer. The repeatability of ChatGPT answers about endometriosis was analyzed by asking each question twice, and the reproducibility of ChatGPT was accepted as scoring the answer to the same question in the same score category.ResultsA total of 91.4% (n = 71) of all FAQs were answered completely, accurately, and sufficiently. ChatGPT had the highest accuracy in the symptom and diagnosis category (94.1%, 16/17 questions) and the lowest accuracy in the treatment category (81.3%, 13/16 questions). Furthermore, of the 40 questions based on the ESHRE endometriosis guidelines, 27 (67.5%) were classified as grade 1, seven (17.5%) as grade 2, and six (15.0%) as grade 3. The reproducibility rate of FAQs in the prevention, symptoms, and diagnosis, and complications categories was the highest (100% for all categories). The reproducibility rate was the lowest for questions based on the ESHRE endometriosis guidelines (70.0%).ConclusionChatGPT accurately and satisfactorily responded to more than 90% of the questions about endometriosis, but to only 67.5% of questions based on the ESHRE endometriosis guidelines.

Artificial Intelligence in the Management of Women with Endometriosis and Adenomyosis: Can Machines Ever Be Worse Than Humans?

“…As to what specifically regards gynecology, Ozgor and Simavi recently analyzed the accuracy of ChatGPT in answering questions about endometriosis. As many as 91% of questions were answered accurately, although among the questions based on the ESHRE endometriosis guidelines [28], accuracy was lower (67.5%) [29]. Considering the speed at which LLM technologies are expanding, and, consequently, how fast they may improve in preciseness and accuracy, these results are encouraging.…”

Toward Clinical Generative AI: Conceptual Framework

“…Conversely, it requires as well ongoing efforts and resources, besides depending on the quality of feedback. A few studies have investigated reproducibility and repeatability [49,50]. In a study [49] involving emergency physicians, six unique prompts were used in conjunction with 61 patient vignettes to assess ChatGPT's ability to assign Canadian Triage and Acuity Scale (CTAS) scores through 10,980 simulated triages.…”

“…Of note, providing more detailed prompts resulted in slightly greater reproducibility but did not significantly improve accuracy. In another study [50] assessing ChatGPT's proficiency in answering frequently asked questions (FAQs) about endometriosis, detailed internet searches were used to compile questions, which were then aligned with the European Society of Human Reproduction and Embryology (ESHRE) guidelines. An experienced gynecologist rated ChatGPT's responses on a scale of 1-4.…”

Evaluating a Large Language Model’s Ability to Answer Clinicians’ Requests for Evidence Summaries

“…Given the ability of generative AI chat tools to quickly produce detailed, fully articulated summaries drawn from a large body of knowledge, evaluating their current performance in responding to clinical questions is critical to understanding how they may eventually be integrated into medical librarians' workflows. Some studies assessing generative AI tools' ability to provide comprehensive and accurate responses to clinical questions have observed that they can produce accurate results [17][18][19][20], particularly for less complex requests [17], although variation in results has been observed among different specialties, tasks, and models investigated [4]. Significant limitations have also been observed, including introduction of both minor and major errors via hallucination or misinterpretation [17,21,22], lack of up-to-date information [23], and limited domain-specific content knowledge [24].…”