Effective User Interaction for High-Recall Retrieval

Zhang, Haotian; Abualsaud, Mustafa; Ghelani, Nimesh; Smucker, Mark D.; Cormack, Gordon V.; Grossman, Maura R.

doi:10.1145/3269206.3271796

Cited by 18 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Looking across all fine-tuning configurations, we see that the top-scoring sentence of each candi-date document alone seems to be a good indicator of document relevance, corroborating the findings of Zhang et al (2018a). Additionally considering the second ranking sentence yields at most a minor gain, and in some cases, adding a third actually causes effectiveness to drop.…”

Section: Resultssupporting

confidence: 73%

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Yilmaz

Yang

Zhang

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

113

121

View full text Add to dashboard Cite

This paper applies BERT to ad hoc document retrieval on news articles, which requires addressing two challenges: relevance judgments in existing test collections are typically provided only at the document level, and documents often exceed the length that BERT was designed to handle. Our solution is to aggregate sentence-level evidence to rank documents. Furthermore, we are able to leverage passage-level relevance judgments fortuitously available in other domains to fine-tune BERT models that are able to capture cross-domain notions of relevance, and can be directly used for ranking news articles. Our simple neural ranking models achieve state-of-the-art effectiveness on three standard test collections.

show abstract

Section: Resultssupporting

confidence: 73%

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Yilmaz

Yang

Zhang

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

113

121

View full text Add to dashboard Cite

show abstract

“…In a recent study from Zhang et al (2018), they conducted a controlled 50users study to evaluate using document excerpts (a single extracted paragraph from document) as relevance feedback in continuous active learning. Participants were asked to find as many relevant documents as possible within one hour using the HiCAL system (Abualsaud et al, 2018b;Zhang et al, 2017;Abualsaud et al, 2018a).…”

Section: Related Workmentioning

confidence: 99%

Evaluating sentence-level relevance feedback for high-recall information retrieval

et al. 2019

Self Cite

View full text Add to dashboard Cite

This study uses a novel simulation framework to evaluate whether the time and effort necessary to achieve high recall using active learning is reduced by presenting the reviewer with isolated sentences, as opposed to full documents, for relevance feedback. Under the weak assumption that more time and effort is required to review an entire document than a single sentence, simulation results indicate that the use of isolated sentences for relevance feedback can yield comparable accuracy and higher efficiency, relative to the state-of-the-art Baseline Model Implementation (BMI) of the AutoTAR Continuous Active Learning ("CAL") method employed in the TREC 2015 and 2016 Total Recall Track.

show abstract

“…Zhang et al [2018c] used a simulation framework to evaluate sentence-level relevance feedback. Zhang et al [2018a] conducted a controlled user study with 50 users to evaluate a retrieval system using the full document or selected paragraph as relevance feedback in CAL. McDonald et al [2018] presented an evaluation of active learning strategies for sensitivity reviews.…”

Section: Technology-assisted Reviewsmentioning

confidence: 99%

Towards Question-based High-recall Information Retrieval

Zou

Kanoulas

2020

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

While continuous active learning algorithms have proven effective in finding most of the relevant documents in a collection, the cost for locating the last few remains high for applications such as Technology-assisted Reviews (TAR). To locate these last few but significant documents efficiently, Zou et al. [2018] have proposed a novel interactive algorithm. The algorithm is based on constructing questions about the presence or absence of entities in the missing relevant documents. The hypothesis made is that entities play a central role in documents carrying key information and that the users are able to answer questions about the presence or absence of an entity in the missing relevance documents. Based on this, a Sequential Bayesian Search-based approach that selects the optimal sequence of questions to ask was devised. In this work, we extend Zou et al. [2018] by (a) investigating the noise tolerance of the proposed algorithm; (b) proposing an alternative objective function to optimize, which accounts for user “erroneous” answers; (c) proposing a method that sequentially decides the best point to stop asking questions to the user; and (d) conducting a small user study to validate some of the assumptions made by Zou et al. [2018]. Furthermore, all experiments are extended to demonstrate the effectiveness of the proposed algorithms not only in the phase of abstract appraisal (i.e., finding the abstracts of potentially relevant documents in a collection) but also finding the documents to be included in the review (i.e., finding the subset of those relevant abstracts for which the article remains relevant). The experimental results demonstrate that the proposed algorithms can greatly improve performance, requiring reviewing fewer irrelevant documents to find the last relevant ones compared to state-of-the-art methods, even in the case of noisy answers. Further, they show that our algorithm learns to stop asking questions at the right time. Last, we conduct a small user study involving an expert reviewer. The user study validates some of the assumptions made in this work regarding the user’s willingness to answer the system questions and the extent of it, as well as the ability of the user to answer these questions.

show abstract

Effective User Interaction for High-Recall Retrieval

Cited by 18 publications

References 22 publications

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Evaluating sentence-level relevance feedback for high-recall information retrieval

Towards Question-based High-recall Information Retrieval

Contact Info

Product

Resources

About