User Preference and Categories for Error Responses in Conversational User Interfaces

Abstract

Error messages are frequent in interactions with Conversational User Interfaces (CUI). Smart speakers respond to about every third user request with an error message. Errors can heavily affect user experience (UX) in interaction with CUI. However, there is limited research on how error responses should be formulated. In this paper, we present a system to study how people classify different categories (acknowledgement of user sentiment, acknowledgement of error and apology) of error messages, and evaluate peoples' preference of error responses with clear categories. The results indicate that if an error response has only one element (i.e. neutral acknowledgement of error, apology or sentiment), responses that acknowledge errors neutrally are preferred by participants. Moreover, we find that when interviewed, participants like error messages to include an apology, an explanation of what went wrong, and a suggestion how to fix the problem in addition to a neutral acknowledgement of an error. Our study has two main contributions: (1) our results inform about the design of error messages and (2) we present a framework for error response categorization and validation.

Chatbot Language - crowdsource perceptions and reactions to dialogue systems to inform dialogue design decisions

“…CBL can be used for a range of research projects, including -designing dialogues of conversational user interfaces, for example by testing one dialogue design versus another in A/B testing, -comparing effects of input modalities, e.g. by running an experiment with a text-based chatbot and comparing user perceptions or behavior to running the same experiment with a speaking chatbot, -investigating effects of errors on user experience (Yuan, Brüggemeier, Hillmann, & Michael, 2020), -examining effects of privacy violations (Sannon, Stoll, DiFranzo, Jung, & Bazarova, 2020) or conversational privacy (Brüggemeier, & Lalone, 2022a), -studying effects of different voices on user behaviour (like male, female and neutral voices (Lee, Nass, & Brave, 2000); or voices with different accents, e.g. British vs. American accents (Tamagawa, Watson, & Kuo, 2011)), -and investigating effects of different languages and cultures (Cambre, & Kulkarni, 2019).…”

Interest identification from browser tab titles: A systematic literature review

“…Given the intriguing and multifaceted research implications but also the strategic business relevance of this subject, studies on how to identify users' interests thrived. Research works focused on several different domains, including, e.g., spreadsheets (Bishop & McDaid, 2008), navigation systems (Wen et al, 2013), and social networks (Antelmi, 2019), but also considered locations of users (Preo t ¸ iuc-Pietro and Cohn, 2013) and preferred user interfaces (Yuan et al, 2020). At the same time, the approaches used for analysing users' behaviours also evolved.…”

Mitigating Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: Exploratory Mixed Methods Experiment

“…Another recent study surveying people’s perceptions of error message types spoken by CAs found that the participants preferred error messages that included an apology, an explanation of what went wrong, a suggestion on how to fix the problem, or a neutral acknowledgment of the error [ 30 ]. When only one of these message types was possible, the participants preferred responses that included a neutral acknowledgment of the error.…”