Success is not Final; Failure is not Fatal – Task Success and User Experience in Interactions with Alexa, Google Assistant and Siri

Abstract

Speech assistants exhibit a high error rate with about one in three user requests resulting in an error. Nonetheless, speech assistants are adopted rapidly with about 1.8 billion users expected in 2021. Given the relatively high task failure rate of speech assistants this may be surprising and raises the question how much user experience (UX) is affected by task success in these devices. We measure user experience with four metrics of UX and evaluate task success in interactions with the speech assistants Alexa, Google Assistant, and Siri. We find that task success only explains between 13% and 30% of the variance of UX. This suggests that a majority of UX is not explained by whether an assistant successfully completes tasks. Moreover, we find that the three assistants do not significantly differ in task success rate, but differ in UX, which supports the conclusion that task success and UX possess limited alignment. We discuss our results and point out limitations and potential future work.

A Review of Subjective Scales Measuring the User Experience of Voice Assistants

“…The significant distinction between Graphical User Interfaces (GUI) and voice user interfaces (VUI) has led to a lack of consensus within the research community, regarding the applicability of scales specifically designed for measuring GUI UX on voice user interface systems such as VA [21], [22]. Nevertheless, on that note, it is important to point out that some studies still utilize scales not specifically designed for voice user interfaces to measure the UX connected to voice assistants [23]- [25] .Therefore, there is a growing interest in developing scales that are specifically tailored for voice user interfaces. However, at present, only a limited number of studies focused on assessing, and analyzing the properties and attributes of the existing scales used for measuring UX of VA [19], and none focus on their development.…”

“…For instance, UEQ and SUS were used by [63] to measure the usability, user experience, and usefulness of the Google Home smart speaker. Furthermore, Attrakdiff was used by [23] to investigate the relationship between task success and user experience of voice assistants, and [61] used it to compare the user experience of ambient voice assistants and button-activated voice assistants. Some studies incorporate multiple scales to carry out their objectives.…”