Prompt engineering for digital mental health: a short review

Priyadarshana, Y. H. P. P.; Senanayake, Ashala; Liang, Zilu; Piumarta, Ian

doi:10.3389/fdgth.2024.1410947

Cited by 3 publications

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Evaluating the Quality of Mental Health Information Generated by Large Language Model Chatbots

Porwal,

Jeenger

2024

Preprint

View full text Add to dashboard Cite

Artificial intelligence (AI) large language models (LLMs) hold great potential to transform psychiatry and mental health care by delivering relevant and tailored mental health information. This study aimed to evaluate the quality of mental health information generated by LLMs by determining their level of accessibility, reliability, and interpreting any bias present. Generative Pre-trained Transformer-4 (GPT-4) (San Francisco, California: Open AI), Gemini 1.5 Flash (Mountain View, California: Google LLC), and Large Language Model Meta AI 3.2 (Llama 3.2) (Menlo Park, California: Meta Inc.) were prompted with 20 questions commonly asked about anxiety and depression. The responses were evaluated using Flesch-Kincaid readability tests to quantify their ease of understanding through grade level and reading ease score measures. The text was subsequently analyzed using a modern DISCERN score to assess the reliability of the health-related information presented. Finally, the LLM responses were evaluated for stigmatizing language using communication and language guidelines for mental health. A significant difference in grade levels and reading ease scores was observed between GPT-4 and Llama (p < 0.01) and Gemini and Llama(p < 0.01), with both GPT-4 and Gemini having higher readability scores. No significant difference was observed in the grade level and reading ease scores between GPT-4 and Gemini (p > 0.05). All three models reported moderate reliability scores but no significant differences were observed (p > 0.05). GPT-4, Gemini, and Llama included stigmatizing phrases in 10%, 15%, and 20% of their responses respectively. These phrases were attributed to descriptions of mental health conditions and substance use; however, no significant differences were observed in the proportions of stigmatizing phrases present across the three models (p > 0.05). Furthermore, comparisons between anxiety and depression responses for each model revealed no significant differences in readability, reliability, or bias. All models demonstrated the ability to generate mental health information that generally satisfied criteria for accessibility, reliability, and minimal bias, with GPT-4 and Gemini reporting higher readability than Llama. However, the lack of additional resources cited in responses and the occasional presence of certain stigmatizing phrases, indicate that additional model training and fine-tuning, specifically for mental health applications, may be necessary before these tools can be deployed on a large scale for mental health care.

show abstract

Evaluating the Quality of Mental Health Information Generated by Large Language Model Chatbots

Porwal,

Jeenger

2024

Preprint

View full text Add to dashboard Cite

show abstract

The Future of Accounting: Sentiment Analysis in AI-Rewritten SEC Filings

Lehner

2024

Preprint

View full text Add to dashboard Cite

Harnessing AI in Anxiety Management: A Chatbot-Based Intervention for Personalized Mental Health Support

Manole,

Cârciumaru,

Brînzaș

et al. 2024

Information

View full text Add to dashboard Cite

Anxiety disorders represent one of the most widespread mental health challenges globally, yet access to traditional therapeutic interventions remains constrained, particularly in resource-limited settings. This study evaluated the effectiveness of an AI-powered chatbot, developed using ChatGPT, in managing anxiety symptoms through evidence-based cognitive-behavioral therapy (CBT) techniques. Fifty participants with mild to moderate anxiety symptoms engaged with the chatbot over two observational phases, each lasting seven days. The chatbot delivered personalized interventions, including mindfulness exercises, cognitive restructuring, and breathing techniques, and was accessible 24/7 to provide real-time support during emotional distress. The findings revealed a significant reduction in anxiety symptoms in both phases, with an average improvement of 21.15% in Phase 1 and 20.42% in Phase 2. Enhanced engagement in Phase 2 suggested the potential for sustained usability and familiarity with the chatbot’s functions. While participants reported high satisfaction with the accessibility and personalization of the chatbot, its inability to replicate human empathy underscored the importance of integrating AI tools with human oversight for optimal outcomes. This study highlights the potential of AI-driven interventions as valuable complements to traditional therapy, providing scalable and accessible mental health support, particularly in regions with limited access to professional services.

show abstract

Prompt engineering for digital mental health: a short review

Cited by 3 publications

References 41 publications

Evaluating the Quality of Mental Health Information Generated by Large Language Model Chatbots

Evaluating the Quality of Mental Health Information Generated by Large Language Model Chatbots

The Future of Accounting: Sentiment Analysis in AI-Rewritten SEC Filings

Harnessing AI in Anxiety Management: A Chatbot-Based Intervention for Personalized Mental Health Support

Contact Info

Product

Resources

About