Understanding the User Experience of Customer Service Chatbots: What Can We Learn from Customer Satisfaction Surveys?

Abstract

Understanding and improving user experience is key to strengthening uptake and realizing the potential of chatbots for customer service. In this paper, we investigate customer satisfaction surveys as a source of insight into such user experience. A total of 5,687 customer satisfaction reports on users' interactions with a customer service chatbot, and the corresponding chatbot interactions, are analyzed. The findings demonstrate that customer satisfaction reports are closely associated with the degree to which the problems motivating users' chatbot interactions are resolved. Furthermore, the findings show substantial variation in the performance of different chatbot intents in terms of customer satisfaction and problem resolution. This implies that user experience varies substantially depending on the problems motivating users to interact with the chatbot. Finally, we identify key characteristics of the intents associated with particularly high or low customer experience, suggesting paths towards efficient improvement of chatbot user experience. Based on the findings, we point to key implications for theory and practice and suggest directions for future research.

Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues

“…Frameworks developed and used for analysis of interaction with other types of chatbots and conversational agents are potentially relevant also to customer service chatbots. Drawing on a literature review of quality assessment in chatbots and conversational agents, Radziwill and Benton [8] proposed a generic framework for assessment of quality attributes such as performance, functionality, human likeness, affective appeal, and accessibility. In the domain of social chatbots, an area of research with increased current research interest following the availability of large language models [31,32], Adiwardana et al [13] proposed a metric for assessing the sensibleness and specificity of chatbot responses and applied this on Google Meena.…”

“…Support for benchmarking and monitoring and comparison of performance over time is of paramount importance in chatbot development and improvement. The importance of benchmarking and comparison is shown in a recent study by Kvale et al [8], where analysis of chatbot performance was used to prioritize chatbot intents for improvement workguiding AI training resources based on the frequency of chatbot intents being triggered and the performance of these relative to other, better performing intents.…”

“…In consequence, service providers need to strengthen capabilities for assessing and monitoring chatbot user experience -to improve identification and evaluation of key drivers of user experience, provide the needed basis for continuous improvement work, and to gain new insight into chatbot users and their evolving patterns of use. While there exists a substantial body of knowledge on the evaluation of spoken dialogue systems [7], there is a need to strengthen capabilities for assessment and monitoring of user experience in current chatbots for customer service [8] which are typically implemented as text-based solutions within company websites or messaging applications [6]. In the present study, we consider qualitative analysis of chatbot dialogues as a means to provide needed capabilities for assessment and monitoring of factors relevant to user experience in such chatbots.…”

Organizational Factors Affecting Successful Implementation of Chatbots for Customer Service

“…A well-designed and implemented chatbot is expected to enrich customer experience and optimize internal operations (PSFK 2018). An inspiring example is the Norwegian telecom provider Telenor's chatbot Telmi, which, in addition to being able to respond to several thousand user intents, also provides support for transactions such service bookings and information about the customer's own subscription (Kvale et al 2020). Such enriched customer experience through chatbots may create a more engaging brand encounter (Chung et al 2018) and strengthen positive brand perceptions (Zarouali et al 2018) and Chatbots and conversational computing has been an area of research for decades.…”

“…Current chatbots for customer service are based on advanced technology support for natural language processing (Kvale et al 2020). Users typically enter their requests in everyday language and the chatbot applies underlying machine learning models to determine the users' intent.…”