Empirical Evaluations of Seed Set Selection Strategies for Predictive Coding

Abstract

Active learning is a popular methodology in text classificationknown in the legal domain as 'predictive coding' or 'Technology Assisted Review' or 'TAR' -due to its potential to minimize the required review effort to build effective classifiers. It is generally assumed that when building a classifier of data for legal purposes (such as production to an opposing party or identification of attorney-client privileged data), the seed set matters less as additional learning rounds are performed, thus in most existing relevant seed set studies the seed set is either built from a random document set or from synthetic documents. However, our recent empirical evaluation on a range of seed set selection strategies demonstrates that the seed set selection strategy can significantly impact predictive coding performance. It is unclear whether that conclusion applies to active learning for predictive coding. In this study, we try to answer that question by using extensive experimentation which examines the impact of popular seed set selection strategies in active learning, within a predictive coding exercise. Additionally, significant research has been devoted to achieving high levels of recall efficiently through continuous active learning strategies when there is an assumption that human review will continue until a certain recall is achieved. However, for reasons such as monetary costs, sensitivity of data (or lack thereof), or time to classify a population, this heavy human lift is often less than ideal for lawyers that are classifying a population for production to an opposing party or classifying a population for attorney-client privilege. Often the strategy is to, instead, minimize the human review effort and to classify a population efficiently with minimal human intervention. In these instances, the selection strategy may be different than what prior research suggests. In this study, we evaluate different active learning strategies against well-researched continuous active learning strategies for the purpose of determining efficient training methods for classifying large populations quickly and precisely. We study how random sampling, keyword models and clustering based seed set selection strategies combined together with top-ranked, uncertain, random, recall inspired, and hybrid active learning document selection strategies affect the performance of active learning for predictive coding. For the purpose of this study, we use the percentage of documents requiring review to reach 75% recall as the 'benchmark' metric to evaluate and compare our approaches. 75% is a commonly used recall threshold in the legal domain when using classifiers to designate documents for production. In most cases we find that seed set selection methods have a minor impact, though they do show significant impact in lower richness data sets or when choosing a top-ranked active learning selection strategy. Our results also show that active learning selection strategies implementing uncertainty, random, or 75% recall selection strategies has the p...