A Multimodal Approach for Improving a Dialogue Agent for Therapeutic Sessions in Psychiatry

Abstract

The number of people with mental health problems is increasing in today’s societies. Unfortunately, there are still not enough experts (psychiatrists, psychotherapists) available. To address this issue, our research team developed a goal-directed therapeutic dialogue system named Terabot to assist psychiatric patients. This system features a voice interface, enabling verbal communication between the patient and the dialogue agent in Polish. Utilizing the RASA framework, the dialogue system is enhanced with text-based emotion and intention recognition. This enables the dialogue system to react “empathically,” i.e., considering the patient’s emotions. The purpose of Terabot is to provide extra support for mental health patients who require additional therapy sessions due to limited access to medical personnel. This will not replace drug treatment but rather serve as additional therapy sessions. Our study consisted of therapy sessions of patients talking to Terabot, conducted at the Institute of Psychiatry and Neurology in Warsaw, Poland. During these sessions, we observed several issues that have led either to interrupting the therapeutic session or worsening the patient’s performance of the relaxation exercise. We suggest addressing these problems by implementing an eye-tracker in our dialogue system to make the dialogue flow more human-like. We propose a feedback loop in which the eye-tracker provides essential data back to the RASA framework. This gives additional information to the framework, and a more appropriate response can be given to the patient. Our main aim is to establish a feedback loop that will likely impact the way the conversation is conducted. Thanks to this, the dialogue system may perform better. As a result, the dialogue agent’s responses can be improved, resulting in a more natural, human-like flow of conversation.