Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues

Abstract

The uptake of chatbots for customer service depends on the user experience. For such chatbots, user experience in particular concerns whether the user is provided relevant answers to their queries and the chatbot interaction brings them closer to resolving their problem. Dialogue data from interactions between users and chatbots represents a potentially valuable source of insight into user experience. However, there is a need for knowledge of how to make use of these data. Motivated by this, we present a framework for qualitative analysis of chatbot dialogues in the customer service domain. The framework has been developed across several studies involving two chatbots for customer service, in collaboration with the chatbot hosts. We present the framework and illustrate its application with insights from three case examples. Through the case findings, we show how the framework may provide insight into key drivers of user experience, including response relevance and dialogue helpfulness (Case 1), insight to drive chatbot improvement in practice (Case 2), and insight of theoretical and practical relevance for understanding chatbot user types and interaction patterns (Case 3). On the basis of the findings, we discuss the strengths and limitations of the framework, its theoretical and practical implications, and directions for future work.

Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service

“…Second, there is likely to be a trade-off between trying to prevent breakdown and conversational efficiency [19]. Our results may suggest that chatbot teams should aim to reduce the chance of breakdown as much as possible.…”

“…While most conversations with well-crafted chatbots for customer service are likely to entail relevant and helpful chatbots responses, such conversations may also involve breakdown. Across a small series of case reports, [19] found chatbots for customer service to provide false positive responses to 14-28 % of user messages and to provide no help in 16-25% of chatbot conversations, depending on available features for breakdown mitigation. Due to the complexity in language interaction, troubles in communication and interpretation are prevalent -even among human conversationalists [49].…”

Exploring the usability and user experience of social media apps through a text mining approach

“…Attempts to discuss the complicated relationship between UX and the concepts of usability in the scientific literature were undertaken by Følstad and Rolfsen (2006) and Sauer et al (2020). They conclude that usability is more objectively measurable than UX.…”

Text Mining Approaches Oriented on Customer Care Efficiency

“…Other software tools can evaluate the chatbot-user dialogue under a qualitative point thus obtaining statistics to be used to augment the efficacy of the customer service (Følstad, et al, 2021). Chatbot is able not only to learn from customers' responses but also to generate inferences about the customers' behaviors, intentions and needs through the dynamic application of knowledge in customer care (Pantano & Pizzi, 2020).…”