Technology Assisted Reviews

Abstract

The goal of a technology-assisted review is to achieve high recall with low human effort. Continuous active learning algorithms have demonstrated good performance in locating the majority of relevant documents in a collection, however their performance is reaching a plateau when 80%-90% of them has been found. Finding the last few relevant documents typically requires exhaustively reviewing the collection. In this paper, we propose a novel method to identify these last few, but significant, documents efficiently. Our method makes the hypothesis that entities carry vital information in documents, and that reviewers can answer questions about the presence or absence of an entity in the missing relevance documents. Based on this we devise a sequential Bayesian search method that selects the optimal sequence of questions to ask. The experimental results show that our proposed method can greatly improve performance requiring less reviewing effort.

Learning to Ask

“…They learn a policy that finds items in a collection using the minimum number of queries. Based on Wen et al [25], Zou et al [30] devised an SBSTAR algorithm to find the last few missing relevant documents in Technology Assisted Reviews by asking "yes" or "no" questions to reviewers. Our QSBPS algorithm differs by performing a cross-user duet training, to learn not only a belief over product relevance but also the reward over the performance of questions, as well as their noise-tolerance.…”

Towards Question-based High-recall Information Retrieval

“…Active learning techniques, which iteratively improve the prediction accuracy by interacting with the reviewers, are considered the state-of-the-art in TAR [O'Mara-Eves et al 2015]. In particular, Grossman [2014, 2017] have proposed a Continuous Active Learning (CAL) algorithm, called Baseline Model Implementation (BMI), which achieves the best performance in a number of high-recall tasks [Grossman et al 2016;Zou et al 2018]. BMI repeatedly trains a logistic regression model to predict the relevance of documents.…”

“…To overcome the above challenge, Zou et al [2018] aim to retrieve these last few missing relevant documents by asking direct questions to reviewers about the information carried in the missing documents, instead of requesting relevance feedback on them. Zou et al [2018] propose a Sequential Bayesian Search-based method for TAR, called SBSTAR. SBSTAR applies CAL up to a certain number of documents reviewed, e.g., 20%-40% of the collection.…”

“…Towards Question-based High-recall Information Retrieval: Locating the Last Few Relevant Documents for Technology-assisted Reviews JIE ZOU and EVANGELOS KANOULAS, University of Amsterdam, The Netherlands While continuous active learning algorithms have proven effective in finding most of the relevant documents in a collection, the cost for locating the last few remains high for applications such as Technology-assisted Reviews (TAR). To locate these last few but significant documents efficiently, Zou et al [2018] have proposed a novel interactive algorithm. The algorithm is based on constructing questions about the presence or absence of entities in the missing relevant documents.…”

“…Based on this, a Sequential Bayesian Search-based approach that selects the optimal sequence of questions to ask was devised. In this work, we extend Zou et al [2018] by (a) investigating the noise tolerance of the proposed algorithm; (b) proposing an alternative objective function to optimize, which accounts for user "erroneous" answers; (c) proposing a method that sequentially decides the best point to stop asking questions to the user; and (d) conducting a small user study to validate some of the assumptions made by Zou et al [2018]. Furthermore, all experiments are extended to demonstrate the effectiveness of the proposed algorithms not only in the phase of abstract appraisal (i.e., finding the abstracts of potentially relevant documents in a collection) but also finding the documents to be included in the review (i.e., finding the subset of those relevant abstracts for which the article remains relevant).…”