Exploring the capacities of ChatGPT: A comprehensive evaluation of its accuracy and repeatability in addressing helicobacter pylori‐related queries

Abstract

BackgroundEducational initiatives on Helicobacter pylori (H. pylori) constitute a highly effective approach for preventing its infection and establishing standardized protocols for its eradication. ChatGPT, a large language model, is a potentially patient‐friendly online tool capable of providing health‐related knowledge. This study aims to assess the accuracy and repeatability of ChatGPT in responding to questions related to H. pylori.Materials and MethodsTwenty‐one common questions about H. pylori were collected and categorized into four domains: basic knowledge, diagnosis, treatment, and prevention. ChatGPT was utilized to individually answer the aforementioned 21 questions. Its responses were independently assessed by two experts on H. pylori. Questions with divergent ratings were resolved by a third reviewer. Cohen's kappa coefficient was calculated to assess the consistency between the scores of the two reviewers.ResultsThe responses of ChatGPT on H. pylori‐related questions were generally satisfactory, with 61.9% marked as “completely correct” and 33.33% as “correct but inadequate.” The repeatability of the responses of ChatGPT to H. pylori‐related questions was 95.23%. Among the responses, those related to prevention (comprehensive: 75%) had the best response, followed by those on treatment (comprehensive: 66.7%), basic knowledge (comprehensive: 60%), and diagnosis (comprehensive: 50%). In the “treatment” domain, 16.6% of the ChatGPT responses were categorized as “mixed with correct or incorrect/outdated data.” However, ChatGPT still lacks relevant knowledge regarding H. pylori resistance and the use of sensitive antibiotics.ConclusionsChatGPT can provide correct answers to the majority of H. pylori‐related queries. It exhibited good reproducibility and delivered responses that were easily comprehensible to patients. Further enhancement of real‐time information updates and correction of inaccurate information will make ChatGPT an essential auxiliary tool for providing accurate H. pylori‐related health information to patients.