A Conversation-Driven Approach for Chatbot Management

Abstract

Managing and evolving a chatbot's content is a laborious process and there is still a lack of standardization. In this context of standardization, the absence of a management process can lead to bad user experiences with a chatbot. This work proposes the Chatbot Management Process, a methodology for content management on chatbot systems. The proposed methodology is based on the experiences acquired with the development of Evatalk, the chatbot for the Brazilian Virtual School of Government. The focus of this methodology is to evolve the chatbot content through the analysis of user interactions, allowing a cyclic and human-supervised process. We divided the proposed methodology into three distinct phases, namely, manage, build, and analyze. Moreover, the proposed methodology presents a clear definition of the roles of the chatbot team. We validate the proposed methodology along with the creation of the Evatalk chatbot, whose amount of interactions was of 22,771 for the 1,698,957 enrolled attendees in the Brazillian Virtual School of Government in 2020. The application of the methodology on Evatalk's chatbot brought positive results: we reduced the chatbot's human hand-off rate from 44.43% to 30.16%, the chatbot's knowledge base examples increased by 160% whilst maintaining a high percentage of confidence in its responses and keeping the user satisfaction collected in conversations stable.

CitizenConnectBot(CCB): The interactive guide to government schemes

“…Complementing this approach, research [2] offers valuable advice for government chatbot designers, delving into the consequences for policy while providing a theoretical framework for understanding interaction model design. Drawing from experiences with Evatalk [3], a chatbot for the Brazilian Virtual School of Government, a cyclic, human-supervised process is introduced in the suggested Chatbot Management Process. Meanwhile, [4] aims to assess the efficacy of rural Maharashtra programs designed to empower women in entrepreneurship.…”

Lhia: A Smart Chatbot for Breastfeeding Education and Recruitment of Human Milk Donors

“…Since we have a more stable version of the chatbot, we plan to put it into production to be used by breastfeeding mothers cared by the HMB of the HU-UFMA. Future plans for further studies include carrying out evaluations related to usability and user experience exploring the production version with mothers and health professionals and, at the same time, applying a process called Conversation-Driven Development (CDD) [57,58]. CDD consists of sharing the chatbot with end users, reviewing regularly generated conversations, taking notes of the conversations, and using them as training data, testing the chatbot to verify that its behavior is as expected, tracking chatbot failures, measuring its performance, and correcting when cases of unsuccessful conversations occur.…”

SkillBot: Towards Data Augmentation using Transformer language model and linguistic evaluation

“…Testing a language model is a difficult task because of the non-discrete nature of the language data. [19] The chatbot application can be assessed using five different evaluation metrics related to user experience, information retrieval process, linguistic capability, business, and technology. We are assessing information retrieval capability in our case by assessing the dialog management system of the chatbot.…”