Use of automated conversational agents in improving young population mental health: a scoping review

Amidst the growing interest in developing task-autonomous AI for automated mental health care, this paper addresses the ethical and practical challenges associated with the issue and proposes a structured framework that delineates levels of autonomy, outlines ethical requirements, and defines beneficial default behaviors for AI agents in the context of mental health support. We also evaluate ten state-of-the-art language models using 16 mental health-related questions designed to reflect various mental health conditions, such as psychosis, mania, depression, suicidal thoughts, and homicidal tendencies. The question design and response evaluations were conducted by mental health clinicians (M.D.s). We find that existing language models are insufficient to match the standard provided by human professionals who can navigate nuances and appreciate context. This is due to a range of issues, including overly cautious or sycophantic responses and the absence of necessary safeguards. Alarmingly, we find that most of the tested models could cause harm if accessed in mental health emergencies, failing to protect users and potentially exacerbating existing symptoms. We explore solutions to enhance the safety of current models. Before the release of increasingly task-autonomous AI systems in mental health, it is crucial to ensure that these models can reliably detect and manage symptoms of common psychiatric disorders to prevent harm to users. This involves aligning with the ethical framework and default behaviors outlined in our study. We contend that model developers are responsible for refining their systems per these guidelines to safeguard against the risks posed by current AI technologies to user mental health and safety.Trigger warningContains and discusses examples of sensitive mental health topics, including suicide and self-harm.

show abstract

Exploring the characteristics of conversational agents in chronic disease management interventions: A scoping review

Uetova,

Hederman,

Ross

et al. 2024

DIGITAL HEALTH

View full text Add to dashboard Cite

Objective With the increasing global burden of chronic diseases, there is the potential for conversational agents (CAs) to assist people in actively managing their conditions. This paper reviews different types of CAs used for chronic condition management, delving into their characteristics and the chosen study designs. This paper also discusses the potential of these CAs to enhance the health and well-being of people with chronic conditions. Methods A search was performed in February 2023 on PubMed, ACM Digital Library, Scopus, and IEEE Xplore. Studies were included if they focused on chronic disease management or prevention and if systems were evaluated on target user groups. Results The 42 selected studies explored diverse types of CAs across 11 health conditions. Personalization varied, with 25 CAs not adapting message content, while others incorporated user characteristics and real-time context. Only 12 studies used medical records in conjunction with CAs for conditions like diabetes, mental health, cardiovascular issues, and cancer. Despite measurement method variations, the studies predominantly emphasized improved health outcomes and positive user attitudes toward CAs. Conclusions The results underscore the need for CAs to adapt to evolving patient needs, customize interventions, and incorporate human support and medical records for more effective care. It also highlights the potential of CAs to play a more active role in helping individuals manage their conditions and notes the value of linguistic data generated during user interactions. The analysis acknowledges its limitations and encourages further research into the use and potential of CAs in disease-specific contexts.

show abstract

Use of automated conversational agents in improving young population mental health: a scoping review

Cited by 4 publications

References 55 publications

Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis

Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis

Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation

Exploring the characteristics of conversational agents in chronic disease management interventions: A scoping review

Contact Info

Product

Resources

About