Evaluation of a spoken dialogue system for controlling a Hifi audio system

Lucas-Cuesta

et al. 2013

Speech Communication

Detecting user affect automatically during real-time conversation is the main challenge towards our greater aim of infusing social intelligence into a natural-language mixed-initiative High-Fidelity (Hi-Fi) audio control spoken dialog agent. In recent years, studies on affect detection from voice have moved on to using realistic, non-acted data, which is subtler. However, it is more challenging to perceive subtler emotions and this is demonstrated in tasks such as labeling and machine prediction. This paper attempts to address part of this challenge by considering the role of user satisfaction ratings and also conversational/dialog features in discriminating contentment and frustration, two types of emotions that are known to be prevalent within spoken human-computer interaction. However, given the laboratory constraints, users might be positively biased when rating the system, indirectly making the reliability of the satisfaction data questionable. Machine learning experiments were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. Our results indicated that standard classifiers were significantly more successful in discriminating the abovementioned emotions and their intensities (reflected by user satisfaction ratings) from annotator data than from user data. These results corroborated that: first, satisfaction data could be used directly as an alternative target variable to model affect, and that they could be predicted exclusively by dialog features. Second, these were only true when trying to predict the abovementioned emotions using annotator's data, suggesting that user bias does exist in a laboratory-led evaluation.

Section: The Corpus Usedmentioning

confidence: 99%

Section: Metrics Of Mixed-initiative Hifi-av2 Spoken Dialogmentioning

confidence: 99%

A satisfaction-based model for affect recognition from conversational features in spoken dialog systems

Lucas-Cuesta

et al. 2013

Speech Communication

“…To model satisfaction we used satisfaction rating as the target and conversational features as predictors, obtained from a corpus collected in a past evaluation [14]. The users involved in the evaluation did not have previous experience in interacting with the HiFi agent, and their participation were not rewarded.…”

Section: Affect Detection Using Satisfaction Ratings (Target) Anmentioning

confidence: 99%

Assessing User Bias in Affect Detection within Context-Based Spoken Dialog Systems

2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing

Casanova-García

et al. 2012

Abstract-This paper presents an empirical evidence of user bias within a laboratory-oriented evaluation of a Spoken Dialog System. Specifically, we addressed user bias in their satisfaction judgements. We question the reliability of this data for modeling user emotion, focusing on contentment and frustration in a spoken dialog system. This bias is detected through machine learning experiments that were conducted on two datasets, users and annotators, which were then compared in order to assess the reliability of these datasets. The target used was the satisfaction rating and the predictors were conversational/dialog features. Our results indicated that standard classifiers were significantly more successful in discriminating frustration and contentment and the intensities of these emotions (reflected by user satisfaction ratings) from annotator data than from user data. Indirectly, the results showed that conversational features are reliable predictors of the two abovementioned emotions.

“…User affect can be reflected in the user satisfaction judgment [1,5,12] and the relationship of affect and satisfaction judgment have been empirically proven in [10,11] and also in our work, which will be further described. To model user affect, we used satisfaction rating as the target and conversational features as predictors, obtained from a corpus collected in a past evaluation [7]. What makes our approach different from others is that we used target and predictor variables whose potentials are often ignored to model affect.…”

Section: Automatic Detection Of Affectmentioning

confidence: 99%

I Feel You: Towards Affect-Sensitive Domotic Spoken Conversational Agents

Lecture Notes in Computer Science

Casanova-García

et al. 2012

Abstract. We describe the work on infusion of emotion into limitedtask autonomous spoken conversational agents (SCAs) situated in the domestic environment, using a Need-inspired task-independent Emotion model (NEMO). In order to demonstrate the generation of affect through the use of the model, we describe the work of integrating it with a naturallanguage mixed-initiative HiFi-control SCA. NEMO and the host system communicates externally, removing the need for the Dialog Manager to be modified as done in most existing dialog systems, in order to be adaptive. We also summarize the work on automatic affect prediction, namely frustration and contentment from dialog features, a non-conventional source, in the attempt of moving towards a more user-centric approach.