Evaluating Voice Skills by Design Guidelines Using an Automatic Voice Crawler

Abstract

Currently, adaptive voice applications supported by voice assistants (VA) are very popular (i.e., Alexa skills and Google Home Actions). Under this circumstance, how to design and evaluate these voice interactions well is very important. In our study, we developed a voice crawler to collect responses from 100 most popular Alexa skills under 10 different categories and evaluated these responses to find out how they comply with 8 selected design guidelines published by Amazon. Our findings show that basic commands support are the most followed ones while those related to personalized interaction are relatively less. There also exists variation in design guidelines compliance across different skill categories. Based on our findings and real skill examples, we offer suggestions for new guidelines to complement the existing ones and propose agendas for future HCI research to improve voice applications' user experiences.

Democratizing Chatbot Debugging: A Computational Framework for Evaluating and Explaining Inappropriate Chatbot Responses

“…• Exploring Framework Generalization Capability: While our results demonstrate promising performance in the context of interview chatbots, the generalizability of these findings to chatbots in diverse domains and with respect to other types of misbehaviors, such as toxic behaviors [6], remains uncertain. Additionally, it is worth investigating the necessary adaptations required to enhance the framework's applicability and generalizability.…”

Designing Effective Interview Chatbots: Automatic Chatbot Profiling and Design Suggestion Generation for Chatbot Debugging

“…Our work on generating design suggestions is also related to various efforts on guiding the design of human-computer interfaces, such as chatbot systems [21] and graphical user interfaces (GUI) [36,68]. For example, Han et al combine domain-specific knowledge together with observational studies to generate rule-based design suggestions for task-oriented chatbots [21]. One of the drawbacks of this approach lies in its inflexibility of adapting design suggestions to changing design goals or dynamic design issues occurring in real time.…”