ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results

Hristidis, Vagelis; Ruggiano, Nicole; Brown, Ellen L.; Ganta, Srivatsava Ranjit; Stewart, Sandra

doi:10.2196/48966

Cited by 43 publications

(9 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can also answer the most frequently asked questions about the COVID-19 pandemic. Its overall responses to queries related to cognitive decline were equivalent to and, at times, more reliable than Google’s [54]. According to Bulck and Moons [55], in comparison to Google search, 40% of the 20 experts (19 nurses; 1 dietitian) considered answers from ChatGPT of greater value, 45% regarded them as equal value and 15% deemed them less valuable.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare

Wang,

Wan,

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: We utilized PubMed, ACM, and IEEE digital libraries as primary sources for this review. We followed the guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRIMSA) to screen and select peer-reviewed research articles that (1) were related to both healthcare applications and conversational LLMs and (2) were published before September 1st, 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare.

show abstract

Section: Resultsmentioning

confidence: 99%

“…Moreover, the readability of an LLM’s response may further accentuate health disparity [54]. LLMs like ChatGPT include texts from scientific websites (e.g.…”

Section: Resultsmentioning

confidence: 99%

A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare

Wang,

Wan,

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Indeed, numerous studies evaluating ChatGPT's performance in medical domains have consistently shown its unreliability in providing clinical information, particularly regarding chronic diseases 36 – 38 , 41 , 42 . Consequently, at present, it appears to be more suitable for addressing basic inquiries, such as those unrelated to clinical aspects.…”

Section: Discussionmentioning

confidence: 99%

“…Initially, the code appeared acceptable, but it ultimately proved to be ineffective and incorrect. Considering the findings from various studies, professional opinions, and discussions, it can be argued that non-professionals, such as informal caregivers of dementia patients, should refrain from using AI bots without consulting a healthcare professional, except for simple questions that do not require clinical knowledge 42 , 43 . In such cases, the purpose of using the bot becomes questionable.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Saeidnia,

Kozak,

Lund

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Many people in the advanced stages of dementia require full-time caregivers, most of whom are family members who provide informal (non-specialized) care. It is important to provide these caregivers with high-quality information to help them understand and manage the symptoms and behaviors of dementia patients. This study aims to evaluate ChatGPT, a chatbot built using the Generative Pre-trained Transformer (GPT) large language model, in responding to information needs and information seeking of such informal caregivers. We identified the information needs of dementia patients based on the relevant literature (22 articles were selected from 2442 retrieved articles). From this analysis, we created a list of 31 items that describe these information needs, and used them to formulate 118 relevant questions. We then asked these questions to ChatGPT and investigated its responses. In the next phase, we asked 15 informal and 15 formal dementia-patient caregivers to analyze and evaluate these ChatGPT responses, using both quantitative (questionnaire) and qualitative (interview) approaches. In the interviews conducted, informal caregivers were more positive towards the use of ChatGPT to obtain non-specialized information about dementia compared to formal caregivers. However, ChatGPT struggled to provide satisfactory responses to more specialized (clinical) inquiries. In the questionnaire study, informal caregivers gave higher ratings to ChatGPT's responsiveness on the 31 items describing information needs, giving an overall mean score of 3.77 (SD 0.98) out of 5; the mean score among formal caregivers was 3.13 (SD 0.65), indicating that formal caregivers showed less trust in ChatGPT's responses compared to informal caregivers. ChatGPT’s responses to non-clinical information needs related to dementia patients were generally satisfactory at this stage. As this tool is still under heavy development, it holds promise for providing even higher-quality information in response to information needs, particularly when developed in collaboration with healthcare professionals. Thus, large language models such as ChatGPT can serve as valuable sources of information for informal caregivers, although they may not fully meet the needs of formal caregivers who seek specialized (clinical) answers. Nevertheless, even in its current state, ChatGPT was able to provide responses to some of the clinical questions related to dementia that were asked.

show abstract

“…Debuting in the latter part of 2022, ChatGPT, an AI chatbot, has demonstrated the capacity to address intricate inquiries spanning a diverse array of subjects, encompassing health and wellness. Moreover, it can adapt its responses according to user inputs [ 5 ].…”

Section: Introductionmentioning

confidence: 99%

Evaluation of the Current State of Chatbots for Digital Health: Scoping Review

Xue,

Zhang,

Zhao

et al. 2023

J Med Internet Res

View full text Add to dashboard Cite

Background Chatbots have become ubiquitous in our daily lives, enabling natural language conversations with users through various modes of communication. Chatbots have the potential to play a significant role in promoting health and well-being. As the number of studies and available products related to chatbots continues to rise, there is a critical need to assess product features to enhance the design of chatbots that effectively promote health and behavioral change. Objective This scoping review aims to provide a comprehensive assessment of the current state of health-related chatbots, including the chatbots’ characteristics and features, user backgrounds, communication models, relational building capacity, personalization, interaction, responses to suicidal thoughts, and users’ in-app experiences during chatbot use. Through this analysis, we seek to identify gaps in the current research, guide future directions, and enhance the design of health-focused chatbots. Methods Following the scoping review methodology by Arksey and O'Malley and guided by the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist, this study used a two-pronged approach to identify relevant chatbots: (1) searching the iOS and Android App Stores and (2) reviewing scientific literature through a search strategy designed by a librarian. Overall, 36 chatbots were selected based on predefined criteria from both sources. These chatbots were systematically evaluated using a comprehensive framework developed for this study, including chatbot characteristics, user backgrounds, building relational capacity, personalization, interaction models, responses to critical situations, and user experiences. Ten coauthors were responsible for downloading and testing the chatbots, coding their features, and evaluating their performance in simulated conversations. The testing of all chatbot apps was limited to their free-to-use features. Results This review provides an overview of the diversity of health-related chatbots, encompassing categories such as mental health support, physical activity promotion, and behavior change interventions. Chatbots use text, animations, speech, images, and emojis for communication. The findings highlight variations in conversational capabilities, including empathy, humor, and personalization. Notably, concerns regarding safety, particularly in addressing suicidal thoughts, were evident. Approximately 44% (16/36) of the chatbots effectively addressed suicidal thoughts. User experiences and behavioral outcomes demonstrated the potential of chatbots in health interventions, but evidence remains limited. Conclusions This scoping review underscores the significance of chatbots in health-related applications and offers insights into their features, functionalities, and user experiences. This study contributes to advancing the understanding of chatbots’ role in digital health interventions, thus paving the way for more effective and user-centric health promotion strategies. This study informs future research directions, emphasizing the need for rigorous randomized control trials, standardized evaluation metrics, and user-centered design to unlock the full potential of chatbots in enhancing health and well-being. Future research should focus on addressing limitations, exploring real-world user experiences, and implementing robust data security and privacy measures.

show abstract

ChatGPT vs Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results

Cited by 43 publications

References 34 publications

A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare

A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Evaluation of the Current State of Chatbots for Digital Health: Scoping Review

Contact Info

Product

Resources

About