What Does ChatGPT Know About Dementia? A Comparative Analysis of Information Quality

Dosso, Jill A.; Kailley, Jaya N.; Robillard, Julie M.

doi:10.3233/jad-230573

Cited by 2 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary aim was to replicate the search behavior of a novice user engaging with ChatGPT for the first time and to assess the quality of the material retrieved from the Large Language Model in comparison to publicly available resources from Alzheimer’s disease-related organizations in North America. The study's findings indicated that users interacting with ChatGPT are likely to access accurate yet superficial information concerning dementia 41 .…”

Section: Discussionmentioning

confidence: 91%

“…Indeed, numerous studies evaluating ChatGPT's performance in medical domains have consistently shown its unreliability in providing clinical information, particularly regarding chronic diseases 36 – 38 , 41 , 42 . Consequently, at present, it appears to be more suitable for addressing basic inquiries, such as those unrelated to clinical aspects.…”

Section: Discussionmentioning

confidence: 99%

“…This represents an area for further exploration and development when deploying intelligent bots in sensitive domains like medicine. Another critical issue is the ability of such chatbots to generate false knowledge due to failure to truly understand the meaning of the language it is processing, also known as “stochastic parroting” 41 . Based on one of the authors (MK) experience gained during discussions with ChatGPT on programming and data science, it appears that in some situations, ChatGPT tries to create knowledge.…”

Section: Discussionmentioning

confidence: 99%

“…Additionally, it is available 24 h a day, 7 days a week, making it a very convenient option for users who need information outside of normal business hours. However, it is important to note that ChatGPT is not infallible and there may be instances where the information provided is not entirely accurate or appropriate 41 . Its answers about medical issues are often imperfect, and in clinical cases, a physician’s expertise and judgment are necessary.…”

Section: Discussionmentioning

confidence: 99%

See 3 more Smart Citations

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Saeidnia,

Kozak,

Lund

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Many people in the advanced stages of dementia require full-time caregivers, most of whom are family members who provide informal (non-specialized) care. It is important to provide these caregivers with high-quality information to help them understand and manage the symptoms and behaviors of dementia patients. This study aims to evaluate ChatGPT, a chatbot built using the Generative Pre-trained Transformer (GPT) large language model, in responding to information needs and information seeking of such informal caregivers. We identified the information needs of dementia patients based on the relevant literature (22 articles were selected from 2442 retrieved articles). From this analysis, we created a list of 31 items that describe these information needs, and used them to formulate 118 relevant questions. We then asked these questions to ChatGPT and investigated its responses. In the next phase, we asked 15 informal and 15 formal dementia-patient caregivers to analyze and evaluate these ChatGPT responses, using both quantitative (questionnaire) and qualitative (interview) approaches. In the interviews conducted, informal caregivers were more positive towards the use of ChatGPT to obtain non-specialized information about dementia compared to formal caregivers. However, ChatGPT struggled to provide satisfactory responses to more specialized (clinical) inquiries. In the questionnaire study, informal caregivers gave higher ratings to ChatGPT's responsiveness on the 31 items describing information needs, giving an overall mean score of 3.77 (SD 0.98) out of 5; the mean score among formal caregivers was 3.13 (SD 0.65), indicating that formal caregivers showed less trust in ChatGPT's responses compared to informal caregivers. ChatGPT’s responses to non-clinical information needs related to dementia patients were generally satisfactory at this stage. As this tool is still under heavy development, it holds promise for providing even higher-quality information in response to information needs, particularly when developed in collaboration with healthcare professionals. Thus, large language models such as ChatGPT can serve as valuable sources of information for informal caregivers, although they may not fully meet the needs of formal caregivers who seek specialized (clinical) answers. Nevertheless, even in its current state, ChatGPT was able to provide responses to some of the clinical questions related to dementia that were asked.

show abstract

Section: Discussionmentioning

confidence: 91%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Saeidnia,

Kozak,

Lund

et al. 2024

Sci Rep

View full text Add to dashboard Cite

show abstract

“…For instance, Hristidis et al (2023) compared ChatGPT with Google search for questions specifically related to dementia and cognitive decline with subpar quality for both systems. In line with this, ChatGPT's knowledge of dementia has been designated as "accurate but shallow" (Dosso et al, 2023). This can potentially be alleviated by finetuning LLMs on medical data.…”

Section: Information Retrievalmentioning

confidence: 99%

Introduction to Large Language Models (LLMs) for dementia care and research

Treder,

Lee,

Tsvetanov

2024

Front. Dement.

View full text Add to dashboard Cite

IntroductionDementia is a progressive neurodegenerative disorder that affects cognitive abilities including memory, reasoning, and communication skills, leading to gradual decline in daily activities and social engagement. In light of the recent advent of Large Language Models (LLMs) such as ChatGPT, this paper aims to thoroughly analyse their potential applications and usefulness in dementia care and research.MethodTo this end, we offer an introduction into LLMs, outlining the key features, capabilities, limitations, potential risks, and practical considerations for deployment as easy-to-use software (e.g., smartphone apps). We then explore various domains related to dementia, identifying opportunities for LLMs to enhance understanding, diagnostics, and treatment, with a broader emphasis on improving patient care. For each domain, the specific contributions of LLMs are examined, such as their ability to engage users in meaningful conversations, deliver personalized support, and offer cognitive enrichment. Potential benefits encompass improved social interaction, enhanced cognitive functioning, increased emotional well-being, and reduced caregiver burden. The deployment of LLMs in caregiving frameworks also raises a number of concerns and considerations. These include privacy and safety concerns, the need for empirical validation, user-centered design, adaptation to the user's unique needs, and the integration of multimodal inputs to create more immersive and personalized experiences. Additionally, ethical guidelines and privacy protocols must be established to ensure responsible and ethical deployment of LLMs.ResultsWe report the results on a questionnaire filled in by people with dementia (PwD) and their supporters wherein we surveyed the usefulness of different application scenarios of LLMs as well as the features that LLM-powered apps should have. Both PwD and supporters were largely positive regarding the prospect of LLMs in care, although concerns were raised regarding bias, data privacy and transparency.DiscussionOverall, this review corroborates the promising utilization of LLMs to positively impact dementia care by boosting cognitive abilities, enriching social interaction, and supporting caregivers. The findings underscore the importance of further research and development in this field to fully harness the benefits of LLMs and maximize their potential for improving the lives of individuals living with dementia.

show abstract

What Does ChatGPT Know About Dementia? A Comparative Analysis of Information Quality

Cited by 2 publications

References 31 publications

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Introduction to Large Language Models (LLMs) for dementia care and research

Contact Info

Product

Resources

About