Evaluation of ChatGPT’s responses to information needs and information seeking of dementia patients

Abstract

Background: Many people in the advanced stages of dementia require full-time caregivers, most of whom are family members who provide informal (non-specialized) care. It is important to provide these caregivers with high-quality information to help them understand and manage the symptoms and behaviors of dementia patients. This study aims to investigate the evaluation of ChatGPT, a chatbot built using the GPT large language model,in responding to information needs and information seeking of such informal caregivers. Methods: We identified the information needs of dementia patients based on the relevant literature (22 articles were selected from 2442 retrieved articles). From this analysis, we created a list of 31 items that describe these information needs, and used them to formulate relevant 118 questions. We then asked these questions to ChatGPT and investigated its responses. In the next phase, we asked 15 informal and 15 formal dementia-patient caregivers to analyze and evaluate these ChatGPT responses, using both quantitative (questionnaire) and qualitative (interview) approaches. Findings: In the interviews conducted, informal caregivers were more positive towards the use of ChatGPT to obtain non-specialized information about dementia compared to formal caregivers. However, ChatGPT struggled to provide satisfactory responses to more specialized (clinical) inquiries. In the questionnaire study, informal caregivers gave higher ratings to ChatGPT's responsiveness on the 31 items describing information needs, giving an overall mean score of 3.77 (SD 0.98) out of 5; the mean score among formal caregivers was 3.13 (SD 0.65), indicating that formal caregivers showed less trust to ChatGPT's responses compared to informal caregivers. Conclusion: ChatGPT’s responses to non-clinical information needs related to dementia patients were generally satisfactory at this stage. As this tool is still under heavy development, it holds promise for providing even higher-quality information in response to information needs, particularly when developed in collaboration with healthcare professionals. Thus, large language models such as ChatGPT can serve as valuable sources of information for informal caregivers, although they may not fully meet the needs of formal caregivers who seek specialized (clinical) answers. Nevertheless, even in its current state, ChatGPT was able to provide responses to some of the clinical questions related to dementia that were asked.

What Does ChatGPT Know About Dementia? A Comparative Analysis of Information Quality

“…Information retrieved from both sources scored poorly on readability. In other work, clinicians judged ChatGPT responses to AD and dementia queries to be satisfactory with some limitations [21], as did formal and informal caregivers of persons living with dementia [22].…”

An Evaluation of General-Purpose AI Chatbots: A Comprehensive Comparative Analysis

“…Weak AI involves developing programs or machines to carry out specific tasks, such as searching for solutions to complex logical formulas with multiple variables, and other decision-making tasks that may surpass human capabilities [4,5]. For example, Weak AI could solve specific tasks such as assistance, self-driving, increasing the quality evaluation processes, offering researchers valuable insights and improving decision-making procedures [13,14].…”

“…One practical application of AI in language understanding is the development of AI chatbots, which gained popularity towards the end of November 2022 with the introduction of the AI chatbot ChatGPT [15,16]. ChatGPT exemplifies the core functionalities of chatbots, including debugging and fixing program code, language translation, grammar checking, conversation support, educational assistance, and creative content generation [13,14,17]. While AI chatbots like ChatGPT can respond to specific queries with reasonable accuracy, they cannot replace the personal interactions, emotional support, and physical assistance that humans offer [13,14].…”