Automatic Online Evaluation of Intelligent Assistants

Abstract

Voice-activated intelligent assistants, such as Siri, Google Now, and Cortana, are prevalent on mobile devices. However, it is challenging to evaluate them due to the varied and evolving number of tasks supported, e.g., voice command, web search, and chat. Since each task may have its own procedure and a unique form of correct answers, it is expensive to evaluate each task individually. This paper is the first attempt to solve this challenge. We develop consistent and automatic approaches that can evaluate different tasks in voice-activated intelligent assistants. We use implicit feedback from users to predict whether users are satisfied with the intelligent assistant as well as its components, i.e., speech recognition and intent classification. Using this approach, we can potentially evaluate and compare different tasks within and across intelligent assistants according to the predicted user satisfaction rates. Our approach is characterized by an automatic scheme of categorizing user-system interaction into task-independent dialog actions, e.g., the user is commanding, selecting, or confirming an action. We use the action sequence in a session to predict user satisfaction and the quality of speech recognition and intent classification. We also incorporate other features to further improve our approach, including features derived from previous work on web search satisfaction prediction, and those utilizing acoustic characteristics of voice requests. We evaluate our approach using data collected from a user study. Results show our approach can accurately identify satisfactory and unsatisfactory sessions.

Using Contextual Information to Understand Searching and Browsing Behavior

“…We also look at the correlation between effort and completion. An obvious finding is that user satisfaction depends on ASR quality which is consistent with previous research [123]. Hence ASR quality is a key component of user satisfaction.…”

“…We proposed three main types of scenarios of use: (1) device control; (2) web search; and (3) structured search dialogue. The scenarios were identified on the basis of three factors: their proportional existence in the logs of a commercial intelligent assistant; the way requests are handled at the intelligent assistant backend (e. g. user requests are redirected to the different services and they serve different interfaces); and the way scenarios were defined in previous works [123]. Next, we investigated: RQ 4.2: How can we measure different aspects of user satisfaction?…”

“…Nowadays, searching behavior is no longer limited to Q, SERP paradigm. For instance, the use of intelligent assistants is bringing new challenges [123,153,245]. One way to improve quality of searching behavior is to predict user satisfaction with SERP upfront.…”

“…There are few studies researching intelligent assistants, and there is only one earlier paper that organizes a user study [123]. Jiang et al [123] focus on simulated tasks for device control, as well as chat and web search, and identify satisfactory and unsatisfactory sessions based on features used in predicting satisfaction on the web, as well as acoustic features of the spoken request. Our work extends this study focusing on a wider range of scenarios of intelligent assistant use, including complex dialogues, and analyzing crucial aspects determining user satisfaction under these different conditions.…”

“…Our first research question is: Based on analysis of the logs of a commercial intelligent assistant; and from previous work [123], we propose three types of scenarios of intelligent assistant use: (1) controlling the device; (2) searching the web; and (3) perform a complex (or 'mission') task in a dialogue interaction. We characterize key aspects of user satisfaction for each of these scenarios.…”