Examining user reviews of conversational systems: a case study of Alexa skills

Abstract

Conversational systems use spoken language to interact with their users. Although conversational systems, such as Amazon Alexa, are becoming common and can provide interesting functionalities, there is li le known about the issues users of these systems face.In this paper, we study user reviews of more than 2,800 Alexa skills to understand the characteristics of the reviews and the issues that they raise. Our results suggest that most skills receive fewer than 50 reviews. Our qualitative study of user reviews using open coding resulted in identifying 16 types of issues in the user reviews. Issues related to content, integration with online services and devices, errors, and regression are the top issues raised by the users. Our results also indicate di erences in volume and types of complaints by users when compared with more traditional mobile applications. We discuss the implication of our results for practitioners and researchers.

IntraVox: A Personalized Human Voice to Support Users with Complex Needs in Smart Homes

“…Despite the above mentioned benefits, some user categories encounter challenges in interacting with voice-enabled personal assistants. For example, Atefi et al [4] discovered that the length of most Alexa Skill commands is greater than 4 words, with some even requesting 8 words. This has led to a complaint by users who tend to find the commands long and di cult.…”