Harnessing Evolution of Multi-Turn Conversations for Effective Answer Retrieval

Abstract

With the improvements in speech recognition and voice generation technologies over the last years, a lot of companies have sought to develop conversation understanding systems that run on mobile phones or smart home devices through natural language interfaces. Conversational assistants, such as Google Assistant™ and Microsoft Cortana™, can help users to complete various types of tasks. This requires an accurate understanding of the user's information need as the conversation evolves into multiple turns. Finding relevant context in a conversation's history is challenging because of the complexity of natural language and the evolution of a user's information need. In this work, we present an extensive analysis of language, relevance, dependency of user utterances in a multi-turn information-seeking conversation. To this aim, we have annotated relevant utterances in the conversations released by the TREC CaST 2019 track. The annotation labels determine which of the previous utterances in a conversation can be used to improve the current one. Furthermore, we propose a neural utterance relevance model based on BERT fine-tuning, outperforming competitive baselines. We study and compare the performance of multiple retrieval models, utilizing different strategies to incorporate the user's context. The experimental results on both classification and retrieval tasks show that our proposed approach can effectively identify and incorporate the conversation context. We show that processing the current utterance using the predicted relevant utterance leads to a 38% relative improvement in terms of nDCG@20. Finally, to foster research in this area, we have released the dataset of the annotations.

Query-to-Session Matching: Do NOT Forget History and Future during Response Selection for Multi-Turn Dialogue Systems

“…However, the global dialogue flow is more like a polymath who is knowledgeable in multiple aspects but may not be as good as the expert in certain aspects. Through our experimental results, we find the combination of these two strategies can further boost the matching performance since they capture the dialogue flow from two different perspectives: (1). the pure and specialized local flow.…”

“…Qu et al [22] propose to understand and characterize how people interact in information-seeking conversation through user intent. Aliannejadi et al [1] explore how to use the utterance relevance in the query through human-annotated labels. Different from these work which need external information and additional human annotations, our proposed query-to-session matching framework only utilizes the built-in history and future utterances in the dialogue sessions.…”

“…We can observe that the response's history and future are all helpful information for the response selection task. To be specific: (1). For the response's history, if the query and response's history both imply the same question, we will have more confidence to conclude that this response can answer the current query well [6].…”

“…The query, the history, and the future are all sequences of utterances. Thus, there are two key problems in this setting: (1). different utterances in the session contribute differently in the query-to-session matching task; (2).…”

“…The model ablation results and the session ablation results indicate the effectiveness of each proposed component and verify the usefulness of the response's history and future. The code is released 1 . The contributions of our paper are:…”