No abstract

This paper examines emotion intensity prediction in dialogs between clients and customer support representatives occurring on Twitter. We focus on a single emotion type, namely, frustration, modelling the user's level of frustration while attempting to predict frustration intensity on the current and next turn, based on the text of turns coming from both dialog participants. A subset of the Kaggle Customer Support on Twitter dataset was used as the modelling data, annotated with per-turn frustration intensity ratings. We propose to represent dialog turns by binary encoded bags of automatically selected keywords to be subsequently used in a machine learning classifier. To assess the classification quality, we examined two different levels of accuracy imprecision tolerance. Our model achieved a level of accuracy significantly higher than a statistical baseline for prediction of frustration intensity for a current turn. However, we did not find the additional information from customer support turns to help predict frustration intensity of the next turn, and the reason for that is possibly the stability of user’s frustration level over the course of the conversation, in other words, the inability of support’s response to exert much influence to user’s initial frustration level.

This paper examines the evolution of emotion intensity in dialogs occurring on Twitter between customer support representatives and clients (“users”). We focus on a single emotion type— frustration, modelling the user's level of frustration (on scale of 0 to 4) for each dialog turn and attempting to predict change of intensity from turn to turn, based on the text of turns from both dialog participants. As the modelling data, we used a subset of the Kaggle Customer Support on Twitter dataset annotated with per-turn frustration intensity ratings. For the modelling, we used a machine learning classifier for which dialog turns were represented by specifically selected bags of words. Since in our experimental setup the prediction classes (i.e., ratings) are not independent, to assess the classification quality, we examined different levels of accuracy imprecision tolerance. We showed that for frustration intensity prediction of actual dialog turns we can achieve a level of accuracy significantly higher than a statistical baseline. However we found that, as the intensity of user’s frustration tends to be stable across turns of the dialog, customer support turns have only a very limited immediate effect on the customer's level of frustration, so using the additional information from customer support turns doesn't help to predict future frustration level.

No abstract