The EMPATHIC Virtual Coach: a demo

Olaso, Javier Mikel; Vázquez, Alain García; Letaifa, Leila Ben; Velasco, Mikel de; Mtibaa, Aymen; Hmani, Mohamed Amine; Petrovska‐Delacrétaz, Dijana; Chollet, Gérard; Montenegro, C.; López-Zorrilla, Asier; Justo, Raquel; Santana, Roberto; Tenorio-Laranga, Jofre; González‐Fraile, Eduardo; Fernández-Ruanova, B.; Cordasco, Gennaro; Esposito, Anna; Gjellesvik, Kristin Beck; Johansen, Anna Torp; Kornes, Maria Stylianou; Pickard, Colin; Glackin, Cornelius; Cahalane, Gary; Buch, Pau; Palmero, Cristina; Escalera, Sérgio; Gordeeva, Olga; Deroo, Olivier; Fernandez, Anaïs; Kyslitska, Daria; Lozano, José A.; Torres, M. Inés; Schlögl, Stephan

doi:10.1145/3462244.3481574

Cited by 4 publications

(8 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also compare these results with the Wizard of Oz (WoZ) data acquisition experiments carried out previously in EMPATHIC [ 71 ], as well as with a fully data-driven coaching chatbot developed with this WoZ data [ 21 ], whenever possible. If the reader is interested in seeing an example of the automatic GROW sessions guided by the proposed dialogue engine, we refer them to [ 70 ], where a video of a session in French subtitled in English is shown.…”

Section: Resultsmentioning

confidence: 99%

“…The differences between countries were largely dependent on the end-users’ connection quality and distance to the servers, located in London. In any way, and even though the system’s performance can be improved in this regard, we find it acceptable for a prototype (the latency can be observed in the EMPATHIC demo [ 70 ]), and there were no big concerns from the end-users either.…”

Section: Resultsmentioning

confidence: 99%

“…This agent is responsible for updating an internal variable storing the last emotion detected in the dashboard. Then, each time the VC has to generate a response the DM assigns an emotion to the voice of the virtual agent according to the user emotion identified [ 2 , 70 ].…”

Section: Dialogue Management To Perform Coaching Strategiesmentioning

confidence: 99%

See 2 more Smart Citations

Dialogue Management and Language Generation for a Robust Conversational Virtual Coach: Validation and User Study

Vázquez

Zorrilla

Olaso

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

Designing human–machine interactive systems requires cooperation between different disciplines is required. In this work, we present a Dialogue Manager and a Language Generator that are the core modules of a Voice-based Spoken Dialogue System (SDS) capable of carrying out challenging, long and complex coaching conversations. We also develop an efficient integration procedure of the whole system that will act as an intelligent and robust Virtual Coach. The coaching task significantly differs from the classical applications of SDSs, resulting in a much higher degree of complexity and difficulty. The Virtual Coach has been successfully tested and validated in a user study with independent elderly, in three different countries with three different languages and cultures: Spain, France and Norway.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Dialogue Management and Language Generation for a Robust Conversational Virtual Coach: Validation and User Study

Vázquez

Zorrilla

Olaso

et al. 2023

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the same vein and though hard to scale up, POMDP-based dialogue managers [28] were developed to cope with the uncertainties related to SDSs, including ASR outputs. Other efforts to include information present in the users' audio signals but absent in the ASR transcription can be found in the area of emotion aware dialogue systems [29], [30]. This kind of systems often include a module devoted to emotion recognition from audio, whose output is employed by the dialogue manager in the decision making step.…”

Section: Related Workmentioning

confidence: 99%

Audio Embedding-Aware Dialogue Policy Learning

Zorrilla

Torres

Cuayáhuitl

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Following the success of Natural Language Processing (NLP) transformers pretrained via self-supervised learning, similar models have been proposed recently for speech processing such as Wav2Vec2, HuBERT and UniSpeech-SAT. An interesting yet unexplored area of application of these models is Spoken Dialogue Systems, where the users' audio signals are typically just mapped to word-level features derived from an Automatic Speech Recogniser (ASR), and then processed using NLP techniques to generate system responses. This paper reports a comprehensive comparison of dialogue policies trained using ASR-based transcriptions and extended with the aforementioned audio processing transformers in the DSTC2 task. Whilst our dialogue policies are trained with supervised and policy-based deep reinforcement learning, they are assessed using both automatic task completion metrics and a human evaluation. Our results reveal that using audio embeddings is more beneficial than detrimental in most of our trained dialogue policies, and that the benefits are stronger for supervised learning than reinforcement learning.

show abstract

“…Research works on VAs with affective capabilities can be found in the literature. Among the most recent examples, we highlight the EMPATHIC Virtual Coach [3] and the Ryan agent [4]. The former [3] modifies the agent's voice based on the user's emotional state, which is inferred from the user's face and the paralinguistic infor-mation embedded in the user's voice recorded during the interaction.…”

Section: Introductionmentioning

confidence: 99%

Coupling Sentiment and Arousal Analysis Towards an Affective Dialogue Manager

Mallol-Ragolta,

Schuller

2024

IEEE Access

View full text Add to dashboard Cite

We present the technologies and host components developed to power a speechbased dialogue manager with affective capabilities. The overall goal is that the system adapts its response to the sentiment and arousal level of the user inferred by analysing the linguistic and paralinguistic information embedded in his or her interaction. A linguistic-based, dedicated sentiment analysis component determines the body of the system response. A paralinguistic-based, dedicated arousal recognition component adjusts the energy level to convey in the affective system response. The sentiment analysis model is trained using the CMU-MOSEI dataset and implements a hierarchical contextual attention fusion network, which scores an Unweighted Average Recall (UAR) of 79.04 % on the test set when tackling the task as a binary classification problem. The arousal recognition model is trained using the MSP-Podcast corpus. This model extracts the Mel-spectrogram representations of the speech signals, which are exploited with a Convolutional Neural Network (CNN) trained from scratch, and scores a UAR of 61.11 % on the test set when tackling the task as a three-class classification problem. Furthermore, we highlight two sample dialogues implemented at the system back-end to detail how the sentiment and arousal inferences are coupled to determine the affective system response. These are also showcased in a proof of concept demonstrator. We publicly release the trained models to provide the research community with off-the-shelf sentiment analysis and arousal recognition tools.

show abstract

The EMPATHIC Virtual Coach: a demo

Cited by 4 publications

References 8 publications

Dialogue Management and Language Generation for a Robust Conversational Virtual Coach: Validation and User Study

Dialogue Management and Language Generation for a Robust Conversational Virtual Coach: Validation and User Study

Audio Embedding-Aware Dialogue Policy Learning

Coupling Sentiment and Arousal Analysis Towards an Affective Dialogue Manager

Contact Info

Product

Resources

About