Evaluating the performance of ChatGPT in answering questions related to pediatric urology

Caglar, Ufuk; Yildiz, Oguzhan; Meric, Arda; Ayranci, Ali; Gelmis, Mucahit; Sarilar, Omer; Ozgor, Faruk

doi:10.1016/j.jpurol.2023.08.003

Cited by 33 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar rate was obtained in the study of Çaglar et al, including 137 questions about pediatric urology; 92% of the questions were answered correctly by ChatGPT ( 5 ). The same study stated that 5.1% of the responses to all questions were correct but insufficient, and 2.9% contained correct information along with misleading information ( 5 ).…”

Section: Discussionsupporting

confidence: 82%

“…Despite having limited access to medical data, ChatGPT performs at the level of a third-year medical student in licensing exams, encouraging discussions on emergency medicine within medicine (3). For example, pediatric urology questions were answered very well in a study conducted using text-based artificial intelligence modeling (5). Although ChatGPT is thought to be promising in producing consistent responses, it is important to determine the accuracy of the medical information it provides.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

How Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?

Tuncer,

Güçlü

2024

Infect Dis Clin Microbiol

View full text Add to dashboard Cite

Objective: The study aimed to investigate the reliability of ChatGPT’s answers to medical questions, including those sourced from patients and guide recommendations. The focus was on evaluating ChatGPT’s accuracy in responding to various types of infectious disease questions. Materials and Methods: The study was conducted using 200 questions sourced from social media, experts, and guidelines related to various infectious diseases like urinary tract infection, pneumonia, HIV, various types of hepatitis, COVID-19, skin infections, and tuberculosis. The questions were arranged for clarity and consistency by excluding repetitive or unclear ones. The answers were based on guidelines from reputable sources like the Infectious Diseases Society of America (IDSA), Centers for Disease Control and Prevention (CDC), European Association for the Study of Liver Disease (EASL) and Joint United Nations Programme on HIV/AIDS (UNAIDS) AIDSinfo. According to the scoring system, completely correct answers were given 1-point, and completely incorrect ones were given 4-points. To assess reproducibility, each question was posed twice on separate computers. Repeatability was determined by the consistency of the answers’ scores. Results: In the study, ChatGPT was posed with 200 questions: 107 from social media platforms and 93 from guidelines. The questions covered a range of topics: urinary tract infections (n=18 questions), pneumonia (n=22), HIV (n=39), hepatitis B and C (n=53), COVID-19 (n=11), skin and soft tissue infections (n=38), and tuberculosis (n=19). The lowest accuracy was 72% for urinary tract infections. ChatGPT answered 92% of social media platform questions correctly (scored 1-point) versus 69% of guideline questions (p=0.001; OR=5.48, 95% CI=2.29-13.11). Conclusion: Artificial intelligence is widely used in the medical field by both healthcare professionals and patients. Although ChatGPT answers questions from social media platforms quite properly, we recommend that healthcare professionals be conscientious when using it. Keywords: infectious disease, Chat-GTP, medical questions

show abstract

Section: Discussionsupporting

confidence: 82%

Section: Introductionmentioning

confidence: 99%

How Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?

Tuncer,

Güçlü

2024

Infect Dis Clin Microbiol

View full text Add to dashboard Cite

show abstract

“…The outputs generated by the language model were generally satisfactory and aligned with current medical guidelines. 69 Similarly, in another study, ChatGPT was tasked with answering frequently asked questions encountered in pediatric cases, such as fever management, appropriate antipyretic dosages, and identification of red flag symptoms. The outputs were deemed moderately accurate and consistent.…”

Section: Cracking the Code—predictive Modeling And Machine Learningmentioning

confidence: 99%

Artificial Intelligence in Pediatrics: Learning to Walk Together

Demirbaş,

Yıldız,

Saygılı

et al. 2024

Turkish Archives of Pediatrics

View full text Add to dashboard Cite

In this era of rapidly advancing technology, artificial intelligence (AI) has emerged as a transformative force, even being called the Fourth Industrial Revolution, along with gene editing and robotics. While it has undoubtedly become an increasingly important part of our daily lives, it must be recognized that it is not an additional tool, but rather a complex concept that poses a variety of challenges. AI, with considerable potential, has found its place in both medical care and clinical research. Within the vast field of pediatrics, it stands out as a particularly promising advancement. As pediatricians, we are indeed witnessing the impactful integration of AI-based applications into our daily clinical practice and research efforts. These tools are being used for simple to more complex tasks such as diagnosing clinically challenging conditions, predicting disease outcomes, creating treatment plans, educating both patients and healthcare professionals, and generating accurate medical records or scientific papers. In conclusion, the multifaceted applications of AI in pediatrics will increase efficiency and improve the quality of healthcare and research. However, there are certain risks and threats accompanying this advancement including the biases that may contribute to health disparities and, inaccuracies. Therefore, it is crucial to recognize and address the technical, ethical, and legal challenges as well as explore the benefits in both clinical and research fields.

show abstract

“…The scoring system, however, has been adapted from similar previous studies. 6 Lastly, the public intelligibility of ChatGPT's answers was not evaluated, which is something that should be assessed in the future.…”

Section: Sources Cetin Et Al Assessed the Quality Of Youtube Videos A...mentioning

confidence: 99%

“…5 Caglar et al found that the answers provided by ChatGPT to patients' questions about pediatric urology and questions prepared based on pediatric urology guidelines were accurate and adequate. 6 Moreover, Gilson et al revealed that ChatGPT was successful in answering medical school examination questions. 7 ChatGPT could also interpret radiological imaging findings with an acceptable error rate.…”

mentioning

confidence: 99%

Accuracy and reproducibility of ChatGPT's free version answers about endometriosis

Ozgor,

Simavi

2023

Intl J Gynecology & Obste

View full text Add to dashboard Cite

ObjectiveTo evaluate the accuracy and reproducibility of ChatGPT's free version answers about endometriosis for the first time.MethodsDetailed internet searches to identify frequently asked questions (FAQs) about endometriosis have been performed. Scientific questions were prepared in accordance with the European Society of Human Reproduction and Embryology (ESHRE) endometriosis guidelines. An experienced gynecologist gave a score of 1–4 for each ChatGPT answer. The repeatability of ChatGPT answers about endometriosis was analyzed by asking each question twice, and the reproducibility of ChatGPT was accepted as scoring the answer to the same question in the same score category.ResultsA total of 91.4% (n = 71) of all FAQs were answered completely, accurately, and sufficiently. ChatGPT had the highest accuracy in the symptom and diagnosis category (94.1%, 16/17 questions) and the lowest accuracy in the treatment category (81.3%, 13/16 questions). Furthermore, of the 40 questions based on the ESHRE endometriosis guidelines, 27 (67.5%) were classified as grade 1, seven (17.5%) as grade 2, and six (15.0%) as grade 3. The reproducibility rate of FAQs in the prevention, symptoms, and diagnosis, and complications categories was the highest (100% for all categories). The reproducibility rate was the lowest for questions based on the ESHRE endometriosis guidelines (70.0%).ConclusionChatGPT accurately and satisfactorily responded to more than 90% of the questions about endometriosis, but to only 67.5% of questions based on the ESHRE endometriosis guidelines.

show abstract

Evaluating the performance of ChatGPT in answering questions related to pediatric urology

Cited by 33 publications

References 11 publications

How Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?

How Reliable is ChatGPT as a Novel Consultant in Infectious Diseases and Clinical Microbiology?

Artificial Intelligence in Pediatrics: Learning to Walk Together

Accuracy and reproducibility of ChatGPT's free version answers about endometriosis

Contact Info

Product

Resources

About