Exploring Conversational Search With Humans, Assistants, and Wizards

Vtyurina, Alexandra; Savenkov, Denis; Agichtein, Eugene; Clarke, Charles L. A.

doi:10.1145/3027063.3053175

Cited by 97 publications

(54 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Vandenberghe introduced the concept of bot personas, which act as off-the-shelf users to allow design teams to interact with rich user data throughout the design process [40]. Others have studied the use of conversational agents in the domains of complex search [2,22,41] or food tracking [14]. These works have shown that conversational agents can improve user experiences and have highlighted the need to further investigate the use of conversational agents in different scenarios.…”

Section: Conversational Agentsmentioning

confidence: 99%

Improving Worker Engagement Through Conversational Microtask Crowdsourcing

Qiu

Gadiraju

Bozzon

2020

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

The rise in popularity of conversational agents has enabled humans to interact with machines more naturally. Recent work has shown that crowd workers in microtask marketplaces can complete a variety of human intelligence tasks (HITs) using conversational interfaces with similar output quality compared to the traditional Web interfaces. In this paper, we investigate the effectiveness of using conversational interfaces to improve worker engagement in microtask crowdsourcing. We designed a text-based conversational agent that assists workers in task execution, and tested the performance of workers when interacting with agents having different conversational styles. We conducted a rigorous experimental study on Amazon Mechanical Turk with 800 unique workers, to explore whether the output quality, worker engagement and the perceived cognitive load of workers can be affected by the conversational agent and its conversational styles. Our results show that conversational interfaces can be effective in engaging workers, and a suitable conversational style has potential to improve worker engagement. Our findings have important implications on workflows and task design with regard to better engaging workers in microtask crowdsourcing marketplaces.

show abstract

Section: Conversational Agentsmentioning

confidence: 99%

Improving Worker Engagement Through Conversational Microtask Crowdsourcing

Qiu

Gadiraju

Bozzon

2020

Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…to clarify ambiguous, faceted or incomplete queries [44]. Asking clarifying questions is especially important in conversational search systems for two reasons: (i) conversation is the most convenient way for natural language interactions and asking questions [22] and (ii) a conversational system can only return a limited number of results, thus being confident about the retrieval performance becomes even more important.…”

mentioning

confidence: 99%

Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Aliannejadi

Zamani

Crestani

et al. 2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

233

290

View full text Add to dashboard Cite

Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s).In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.

show abstract

“…They thus provided a framework for understanding the human-computer interaction from an agent's point of view. Vtyurina et al [37] listed the important factors to consider while designing a conversational assistant. One of their key findings was that it is essential to maintain the conversational context which is one of the focal points of our paper.…”

Section: Related Workmentioning

confidence: 99%

Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval

Aliannejadi

Chakraborty

Ríssola

et al. 2020

Proceedings of the 2020 Conference on Human Information Interaction and Retrieval

View full text Add to dashboard Cite

With the improvements in speech recognition and voice generation technologies over the last years, a lot of companies have sought to develop conversation understanding systems that run on mobile phones or smart home devices through natural language interfaces. Conversational assistants, such as Google Assistant™ and Microsoft Cortana™, can help users to complete various types of tasks. This requires an accurate understanding of the user's information need as the conversation evolves into multiple turns. Finding relevant context in a conversation's history is challenging because of the complexity of natural language and the evolution of a user's information need. In this work, we present an extensive analysis of language, relevance, dependency of user utterances in a multi-turn information-seeking conversation. To this aim, we have annotated relevant utterances in the conversations released by the TREC CaST 2019 track. The annotation labels determine which of the previous utterances in a conversation can be used to improve the current one. Furthermore, we propose a neural utterance relevance model based on BERT fine-tuning, outperforming competitive baselines. We study and compare the performance of multiple retrieval models, utilizing different strategies to incorporate the user's context. The experimental results on both classification and retrieval tasks show that our proposed approach can effectively identify and incorporate the conversation context. We show that processing the current utterance using the predicted relevant utterance leads to a 38% relative improvement in terms of nDCG@20. Finally, to foster research in this area, we have released the dataset of the annotations.

show abstract

Exploring Conversational Search With Humans, Assistants, and Wizards

Cited by 97 publications

References 11 publications

Improving Worker Engagement Through Conversational Microtask Crowdsourcing

Improving Worker Engagement Through Conversational Microtask Crowdsourcing

Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval

Contact Info

Product

Resources

About