Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?

Abstract

Chatbots are becoming increasingly popular as a human-computer interface. The traditional best practices normally applied to User Experience (UX) design cannot easily be applied to chatbots, nor can conventional usability testing techniques guarantee accuracy. WeightMentor is a bespoke self-help motivational tool for weight loss maintenance. This study addresses the following four research questions: How usable is the WeightMentor chatbot, according to conventional usability methods?; To what extend will different conventional usability questionnaires correlate when evaluating chatbot usability?; And how do they correlate to a tailored chatbot usability survey score?; What is the optimum number of users required to identify chatbot usability issues?; How many task repetitions are required for a first-time chatbot users to reach optimum task performance (i.e. efficiency based on task completion times)? This paper describes the procedure for testing the WeightMentor chatbot, assesses correlation between typical usability testing metrics, and suggests that conventional wisdom on participant numbers for identifying usability issues may not apply to chatbots. The study design was a usability study. WeightMentor was tested using a predetermined usability testing protocol, evaluating ease of task completion, unique usability errors and participant opinions on the chatbot (collected using usability questionnaires). WeightMentor usability scores were generally high, and correlation between questionnaires was strong. The optimum number of users for identifying chatbot usability errors was 26, which challenges previous research. Chatbot users reached optimum proficiency in tasks after just one repetition. Usability test outcomes confirm what is already known

Virtual Assistant: A Multi-paradigm Dialog Workflow System for Visitor Registration During a Pandemic Situation

“…Eight testers were recruited and tested in total 16 modalities. This is lower than the recommended 26 for conversational interface evaluation [4] and could be a threat to liability.…”

“…14 test modality tests with seven testers were executed. This is lower than the recommended 26 test subjects for conversational interface evaluation by Holmes et al [4] and could be a threat to liability. However, we assume that our results would be significant due to the directional nature of our hypothesis.…”

“…This study builds a prototype of a system which is evaluated using usability methods proposed for conversational interfaces [4]. We design the VVA as a non-intrusive actor of communication by the means of a conversational interface as a chatbot-kind of operated voice-recognition interface (VOI) with a supportive display, to facilitate contactless registration of business visitors (Fig.…”

“…Based on inspiration from Holmes et al [4], we will evaluate the interfaces using quantitative statistics based on measurable metrics and a questionnaire, as well as a qualitative interview of the study group.…”

“…Holmes et al [4] conducted experiments to contribute to the understanding whether we can use conventional methods to assess conversational interfaces. The usability tests were based on observational research, with think-aloud testing protocol and questionnaires.…”