Responses of Five Different Artificial Intelligence Chatbots to the Top Searched Queries About Erectile Dysfunction: A Comparative Analysis

Abstract

The aim of the study is to evaluate and compare the quality and readability of responses generated by five different artificial intelligence (AI) chatbots—ChatGPT, Bard, Bing, Ernie, and Copilot—to the top searched queries of erectile dysfunction (ED). Google Trends was used to identify ED-related relevant phrases. Each AI chatbot received a specific sequence of 25 frequently searched terms as input. Responses were evaluated using DISCERN, Ensuring Quality Information for Patients (EQIP), and Flesch-Kincaid Grade Level (FKGL) and Reading Ease (FKRE) metrics. The top three most frequently searched phrases were “erectile dysfunction cause”, “how to erectile dysfunction,” and “erectile dysfunction treatment.” Zimbabwe, Zambia, and Ghana exhibited the highest level of interest in ED. None of the AI chatbots achieved the necessary degree of readability. However, Bard exhibited significantly higher FKRE and FKGL ratings (p = 0.001), and Copilot achieved better EQIP and DISCERN ratings than the other chatbots (p = 0.001). Bard exhibited the simplest linguistic framework and posed the least challenge in terms of readability and comprehension, and Copilot’s text quality on ED was superior to the other chatbots. As new chatbots are introduced, their understandability and text quality increase, providing better guidance to patients.

Evaluation of ChatGPT's Responses to Google Searches About Dry Eye (Preprint)

“…Temel et al reported that content created by chatGPT regarding spinal cord injury had substantial difficulties with quality 6 . Şahin et al compared 5 different AI chatbots about erectile dysfunction and found that none of the chatbots had the required level of readability and quality25 . Similar to all this research, our study also demonstrated that content about DED created by ChatGPT has significant quality issues.…”

Letter to the Editor of the Journal of Medical Systems: Regarding “Responses of Five Different Artificial Intelligence Chatbots to the Top Searched Queries About Erectile Dysfunction: A Comparative Analysis”

“…In the ‘Materials and Methods’ section, the authors detail their analysis of these responses using various readability scales, including DISCERN, Ensuring Quality Information for Patients (EQIP), and the Flesch-Kincaid Grade Level (FKGL) and Reading Ease (FKRE). Additionally, they disclose the names of the five AI-based chatbots used to generate the responses under analysis [ 1 ].…”