The Role of Trust in Proactive Conversational Assistants

Kraus, Matthias; Wagner, Nicolas; Callejas, Zoraida; Minker, Wolfgang

doi:10.1109/access.2021.3103893

Cited by 31 publications

(15 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So a CUI can be assertive and take the lead when it is appropriate. For example, it has been shown that proactive dialogue strategies have an effect on the human-computer trust relationships depending on context [22] and user-specific information [21]. Furthermore, proactive explanations are able to foster and maintain trust as a response to system breakdowns [38].…”

Section: Position Paper Topicsmentioning

confidence: 99%

Ethics of Conversational User Interfaces

Lee

Sin

Laban

et al. 2022

CHI Conference on Human Factors in Computing Systems Extended Abstracts

Self Cite

View full text Add to dashboard Cite

Building on the prior workshops on conversational user interfaces (CUIs) [2,40], we tackle the topic of ethics of CUIs at CHI 2022. Though commercial CUI developments continue to rapidly advance, our scholarly dialogue on ethics of CUIs is underwhelming. The CUI community has implicitly been concerned with ethics, yet making it central to the growing body of work thus far has not been adequately done. Since ethics is a far-reaching topic, perspectives from philosophy, design, and engineering domains are integral to our CUI research community. For instance, philosophical traditions, e.g., deontology or virtue ethics, can guide ethical concepts that are relevant for CUIs, e.g., autonomy or trust. The practice of design through approaches like value sensitive design can inform how CUIs should be developed. Ethics comes into play with technical contributions, e.g., privacy-preserving data sharing between conversational systems. By considering such multidisciplinary angles, we come to a special topic of interest that ties together philosophy, design, and engineering: conversational disclosure, e.g., sharing personal information, transparency, e.g., as how to transparently convey relevant information in a conversational manner, and vulnerability of diverse user groups that should be taken into consideration. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI); Haptic devices; User studies.

show abstract

Section: Position Paper Topicsmentioning

confidence: 99%

Ethics of Conversational User Interfaces

Lee

Sin

Laban

et al. 2022

CHI Conference on Human Factors in Computing Systems Extended Abstracts

Self Cite

View full text Add to dashboard Cite

show abstract

“…To obtain the emotional tag from our bi-LSTM layer, we concatenated for each embedding of a frame the outputs obtained from the analysis performed in each specific direction (see Equation (1) in which || corresponds to the concatenation operator and L to the size of each LSTM).…”

Section: Feature Extractionmentioning

confidence: 99%

“…By analyzing individuals' behavior, it is also possible to detect a loss of trust or changes in emotions. This capability lets that specific system, such as Conversational Systems and Embodied Conversational Agents (ECAs) [1,2], react to these events and adapt their actions to improve interactions or modify the dialogue contents, tone, or facial expressions to create a better socio-affective user experience [3].…”

Section: Introductionmentioning

confidence: 99%

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Jiménez

Griol

Callejas

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

show abstract

“…To obtain the emotional tag from the Bi-LSTM layers, we concatenated the embeddings of the outputs of each specific direction (see Equation (1) in which || corresponds to the concatenation operator and L to the size of each LSTM).…”

Section: Static Vs Sequential Modelsmentioning

confidence: 99%

“…With the help of an emotion recognizer, other systems could detect loss of trust or changes in emotions by monitoring people's conduct. This capability will help specific systems such as Embodied Conversational Agents (ECAs) [1,2] to react to these events and adapt their decisions to improve conversations by adjusting their tone or facial expressions to create a better socio-affective user experience [3].…”

Section: Introductionmentioning

confidence: 99%

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

et al. 2021

Self Cite

View full text Add to dashboard Cite

Emotion recognition is attracting the attention of the research community due to its multiple applications in different fields, such as medicine or autonomous driving. In this paper, we proposed an automatic emotion recognizer system that consisted of a speech emotion recognizer (SER) and a facial emotion recognizer (FER). For the SER, we evaluated a pre-trained xlsr-Wav2Vec2.0 transformer using two transfer-learning techniques: embedding extraction and fine-tuning. The best accuracy results were achieved when we fine-tuned the whole model by appending a multilayer perceptron on top of it, confirming that the training was more robust when it did not start from scratch and the previous knowledge of the network was similar to the task to adapt. Regarding the facial emotion recognizer, we extracted the Action Units of the videos and compared the performance between employing static models against sequential models. Results showed that sequential models beat static models by a narrow difference. Error analysis reported that the visual systems could improve with a detector of high-emotional load frames, which opened a new line of research to discover new ways to learn from videos. Finally, combining these two modalities with a late fusion strategy, we achieved 86.70% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. Results demonstrated that these modalities carried relevant information to detect users’ emotional state and their combination allowed to improve the final system performance.

show abstract

The Role of Trust in Proactive Conversational Assistants

Cited by 31 publications

References 67 publications

Ethics of Conversational User Interfaces

Ethics of Conversational User Interfaces

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

Contact Info

Product

Resources

About