Measuring User Satisfaction on Smart Speaker Intelligent Assistants Using Intent Sensitive Query Embeddings

Hashemi, Seyyed Hadi; Williams, Kyle; Kholy, Ahmed El; Zitouni, Imed; Crook, Paul A.

doi:10.1145/3269206.3271802

Cited by 36 publications

(15 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Giannopoulos et al [26] proposed a clientcentered intent-aware query framework to shield user data privacy in personalized web search [25]. Hashemi et al [27] proposed a multiple intent model to infer users' behavioral intention from America Online (AOL) search query log. Peng et al [28] proposed a structural equation-based model to discover the factors of discontinuance intention towards social network sites (SNS) concerning autonomous and controlled motivations.…”

Section: Behavioral Intentionsmentioning

confidence: 99%

Social media intention mining for sustainable information systems: categories, taxonomy, datasets and challenges

Rashid

Farooq

Abid

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Intention mining is a promising research area of data mining that aims to determine end-users’ intentions from their past activities stored in the logs, which note users’ interaction with the system. Search engines are a major source to infer users’ past searching activities to predict their intention, facilitating the vendors and manufacturers to present their products to the user in a promising manner. This area has been consistently getting pertinence with an increasing trend for online purchasing. Noticeable research work has been accomplished in this area for the last two decades. There is no such systematic literature review available that provides a comprehensive review in intension mining domain to the best of our knowledge. This article presents a systematic literature review based on 109 high-quality research papers selected after rigorous screening. The analysis reveals that there exist eight prominent categories of intention. Furthermore, a taxonomy of the approaches and techniques used for intention mining have been discussed in this article. Similarly, six important types of data sets used for this purpose have also been discussed in this work. Lastly, future challenges and research gaps have also been presented for the researchers working in this domain.

show abstract

Section: Behavioral Intentionsmentioning

confidence: 99%

Social media intention mining for sustainable information systems: categories, taxonomy, datasets and challenges

Rashid

Farooq

Abid

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

“…Other researchers have proposed ways to measure new signals. For instance, Hashemi et al [29] proposed user intent as an original signal for measuring user satisfaction. Moreover, the personification of Alexa is linked to a higher level of user satisfaction due to increased social interactions [43].…”

Section: Ux Of Smart Speakersmentioning

confidence: 99%

“Alexa, Are You Spying on Me?”: Exploring the Effect of User Experience on the Security and Privacy of Smart Speaker Users

Chalhoub

Fléchais

2020

HCI for Cybersecurity, Privacy and Trust

View full text Add to dashboard Cite

Smart speakers are useful and convenient, but they are associated with numerous security and privacy threats. We conducted thirteen interviews with users of smart speakers to explore the effect of user experience (UX) factors on security and privacy. We analyzed the data using Grounded Theory and validated our results with a qualitative meta-synthesis. We found that smart speaker users lack privacy concerns towards smart speakers, which prompts them to trade their privacy for convenience. However, various trigger points such as negative experiences evoke security and privacy needs. When such needs emerge, existing security and privacy features were not found to be user-friendly which resulted in compensatory behavior. We used our results to propose a conceptual model demonstrating UX's effect on risk, perceptions and balancing behavior. Finally, we concluded our study by recommending user-friendly security and privacy features for smart speakers.

show abstract

“…To represent context, we draw on active prior work on representation learning using unsupervised feature discovery to predict user satisfaction. One recent work proposed a query representation learning technique with intent-sensitive word embeddings, and showed that modifications to improve query representation can improve overall model performance [16]. Another recent work introduced a model that can detect egregious conversations using textual representations, and addressed how this technique can be applied to an automated evaluation scheme [28].…”

Section: Related Workmentioning

confidence: 99%

“…Evaluating intelligent assistants is a challenging task, and has been an active area of research. For example, recent studies identified particular patterns of interactions which tend to contribute to final user satisfaction (e.g., [16,21,28]), and new behavioral metrics such as conversational depth and topic diversity have been proposed to systematically evaluate user experience in conversational systems [20,26,32]. However, as we will show, these metrics do not directly correspond to actual subjective and immediate user satisfaction with the conversation.…”

mentioning

confidence: 90%

Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

Choi

Ahmadvand

Agichtein

2019

Proceedings of the 28th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Predicting user satisfaction in conversational systems has become critical, as spoken conversational assistants operate in increasingly complex domains. Online satisfaction prediction (i.e., predicting satisfaction of the user with the system after each turn) could be used as a new proxy for implicit user feedback, and offers promising opportunities to create more responsive and effective conversational agents, which adapt to the user's engagement with the agent. To accomplish this goal, we propose a conversational satisfaction prediction model specifically designed for open-domain spoken conversational agents, called ConvSAT. To operate robustly across domains, ConvSAT aggregates multiple representations of the conversation, namely the conversation history, utterance and response content, and system-and user-oriented behavioral signals. We first calibrate ConvSAT performance against state of the art methods on a standard dataset (Dialogue Breakdown Detection Challenge) in an online regime, and then evaluate ConvSAT on a large dataset of conversations with real users, collected as part of the Alexa Prize competition. Our experimental results show that ConvSAT significantly improves satisfaction prediction for both offline and online setting on both datasets, compared to the previously reported stateof-the-art approaches. The insights from our study can enable more intelligent conversational systems, which could adapt in real-time to the inferred user satisfaction and engagement.

show abstract

Measuring User Satisfaction on Smart Speaker Intelligent Assistants Using Intent Sensitive Query Embeddings

Cited by 36 publications

References 43 publications

Social media intention mining for sustainable information systems: categories, taxonomy, datasets and challenges

Social media intention mining for sustainable information systems: categories, taxonomy, datasets and challenges

“Alexa, Are You Spying on Me?”: Exploring the Effect of User Experience on the Security and Privacy of Smart Speaker Users

Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

Contact Info

Product

Resources

About