Query Resolution for Conversational Search with Limited Supervision

Voskarides, Nikos; Li, Dan; Ren, Pengjie; Kanoulas, Evangelos; Rijke, Maarten de

doi:10.1145/3397271.3401130

Cited by 79 publications

(43 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning based query expansion methods tackle CQU of Equation 3.9 by defining a binary term classification problem: for each term appearing in dialog history 𝑞 ∈ H 𝑖 , decide whether to add it to the current query 𝑄 𝑖 . That is, S * consists of all the terms selected by a binary term classifier learned from training data [e.g., Voskarides et al, 2020;Cao et al, 2008;Xiong and Callan, 2015].…”

Section: Machine Learning Based Query Expansion Methodsmentioning

confidence: 99%

“…If the component augments the query before retrieval, then it can be evaluated using the retrieval-based evaluation in the previous section. Using TREC CAsT data, several studies employed methods that combine the current query with previous queries, with the goal that the augmented query will yield improved search results [Yang et al, 2019;Vakulenko et al, 2021;Voskarides et al, 2020]. These methods for query expansion and query rewriting are described in more detail in Chapter 3, along with other methods that use recognition and resolution to better process the text of the query and the candidate answers.…”

Section: Evaluating Non-retrieval Componentsmentioning

confidence: 99%

“…Although the above procedure is noisy and can result in adding terms to 𝑄 𝑖 that are non-relevant, or adding too few relevant terms 𝑄 𝑖 , Voskarides et al [2020] report good performance on the TREC CAsT 2019 dataset, where the majority of expansion terms in the oracle manual de-contextualized queries (see the example in Table 3.2) are indeed from the previous query turns. However, it remains to be validated whether the observation and good performance hold on other conversational search benchmarks or in real-world scenarios.…”

Section: Generating Distance Supervisionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Approaches to Conversational AI

Gao

Galley

2019

FNT in Information Retrieval

170

View full text Add to dashboard Cite

A conversational information retrieval (CIR) system is an information retrieval (IR) system with a conversational interface which allows users to interact with the system to seek information via multi-turn conversations of natural language, in spoken or written form. Recent progress in deep learning has brought tremendous improvements in natural language processing (NLP) and conversational AI, leading to a plethora of commercial conversational services that allow naturally spoken and typed interaction, increasing the need for more human-centric interactions in IR. As a result, we have witnessed a resurgent interest in developing modern CIR systems in both research communities and industry. This book surveys recent advances in CIR, focusing on neural approaches that have been developed in the last few years. This book is based on the authors' tutorial at SIGIR'2020 [Gao et al., 2020b], with IR and NLP communities as the primary target audience. However, audiences with other background, such as machine learning and human-computer interaction, will also find it an accessible introduction to CIR. We hope that this book will prove a valuable resource for students, researchers, and software developers. This manuscript is a working draft. Comments are welcome.

show abstract

Section: Machine Learning Based Query Expansion Methodsmentioning

confidence: 99%

Section: Evaluating Non-retrieval Componentsmentioning

confidence: 99%

Section: Generating Distance Supervisionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Approaches to Conversational AI

Gao

Galley

2019

FNT in Information Retrieval

170

View full text Add to dashboard Cite

show abstract

“…The current speech recognition technique was hard to satisfy users’ need (Guy, 2018). Different classifiers for voice queries were proposed to understand user intent accurately (Mukherjee et al, 2013; Qu et al, 2019; Voskarides, 2020). And playing with a voice assistant without a body and figure was perceived as a less entertainment experience comparing with a robot (Pollmann et al, 2020).…”

Section: Literature Reviewmentioning

confidence: 99%

Why Do You Trust Siri?: The Factors Affecting Trustworthiness of Intelligent Personal Assistant

Huang

2021

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Trust greatly contributes to human-AI collaboration, however, human's trust to IPA is hard to establish and lacks exploration. The purpose of this paper is to recognize the factors that affect the trustworthiness of IPA. 358 questionnaires were analyzed by PLS-SEM to construct the model, while thematic analysis was used to discover expectance of IPA. Chi-square tests and T-test were used to distinguish the difference between two user groups. Three factors that capability of system, personality of agent, and availability of interface have a significant impact on the trustworthiness of IPA. The capability of system is the most essential as the threshold with users' plenty of expectations. Most users pay less attention to the availability of interface and the personality of agent has a great impact on the trustworthiness of IPA. The factors found enrich the trusted AI research and inspire insights of design of IPA.

show abstract

“…We fine-tune a pre-trained BERT model [49] on the relevance classification datasets of the TREC Web 2009, 2010 and 2011 tracks [29,30,150], and then take each query-document pair as the input and output vector as it semantic representation. Fine-tuning a pre-trained BERT model on downstream tasks like relevance classification have been widely proven effective [164]. The three web tracks are used as the fine-tuning dataset because they share the same ClueWeb09 document collection with the crowdsourcing datasets in our experiments in Section 4.5.…”

Section: Task Representationmentioning

confidence: 99%

Effective collection construction for information retrieval evaluation and optimization

2020

SIGIR Forum

Self Cite

View full text Add to dashboard Cite

The availability of test collections in Cranfield paradigm has significantly benefited the development of models, methods and tools in information retrieval. Such test collections typically consist of a set of topics, a document collection and a set of relevance assessments. Constructing these test collections requires effort of various perspectives such as topic selection, document selection, relevance assessment, and relevance label aggregation etc. The work in the thesis provides a fundamental way of constructing and utilizing test collections in information retrieval in an effective, efficient and reliable manner. To that end, we have focused on four aspects. We first study the document selection issue when building test collections. We devise an active sampling method for efficient large-scale evaluation [Li and Kanoulas, 2017]. Different from past sampling-based approaches, we account for the fact that some systems are of higher quality than others, and we design the sampling distribution to over-sample documents from these systems. At the same time, the estimated evaluation measures are unbiased, and assessments can be used to evaluate new, novel systems without introducing any systematic error. Then a natural further step is determining when to stop the document selection and assessment procedure. This is an important but understudied problem in the construction of test collections. We consider both the gain of identifying relevant documents and the cost of assessing documents as the optimization goals. We handle the problem under the continuous active learning framework by jointly training a ranking model to rank documents, and estimating the total number of relevant documents in the collection using a "greedy" sampling method [Li and Kanoulas, 2020]. The next stage of constructing a test collection is assessing relevance. We study how to denoise relevance assessments by aggregating from multiple crowd annotation sources to obtain high-quality relevance assessments. This helps to boost the quality of relevance assessments acquired in a crowdsourcing manner. We assume a Gaussian process prior on query-document pairs to model their correlation. The proposed model shows good performance in terms of interring true relevance labels. Besides, it allows predicting relevance labels for new tasks that has no crowd annotations, which is a new functionality of CrowdGP. Ablation studies demonstrate that the effectiveness is attributed to the modelling of task correlation based on the axillary information of tasks and the prior relevance information of documents to queries. After a test collection is constructed, it can be used to either evaluate retrieval systems or train a ranking model. We propose to use it to optimize the configuration of retrieval systems. We use Bayesian optimization approach to model the effect of a δ -step in the configuration space to the effectiveness of the retrieval system, by suggesting to use different similarity functions (covariance functions) for continuous and categorical values, and examine their ability to effectively and efficiently guide the search in the configuration space [Li and Kanoulas, 2018]. Beyond the algorithmic and empirical contributions, work done as part of this thesis also contributed to the research community as the CLEF Technology Assisted Reviews in Empirical Medicine Tracks in 2017, 2018, and 2019 [Kanoulas et al., 2017, 2018, 2019]. Awarded by: University of Amsterdam, Amsterdam, The Netherlands. Supervised by: Evangelos Kanoulas. Available at: https://dare.uva.nl/search?identifier=3438a2b6-9271-4f2c-add5-3c811cc48d42.

show abstract

Query Resolution for Conversational Search with Limited Supervision

Cited by 79 publications

References 44 publications

Neural Approaches to Conversational AI

Neural Approaches to Conversational AI

Why Do You Trust Siri?: The Factors Affecting Trustworthiness of Intelligent Personal Assistant

Effective collection construction for information retrieval evaluation and optimization

Contact Info

Product

Resources

About