A utility-theoretic ranking method for semi-automated text classification

Lecture Notes in Computer Science

Ounis

2017

Abstract. Government documents must be reviewed to identify any sensitive information they may contain, before they can be released to the public. However, traditional paper-based sensitivity review processes are not practical for reviewing born-digital documents. Therefore, there is a timely need for automatic sensitivity classification techniques, to assist the digital sensitivity review process. However, sensitivity is typically a product of the relations between combinations of terms, such as who said what about whom, therefore, automatic sensitivity classification is a difficult task. Vector representations of terms, such as word embeddings, have been shown to be effective at encoding latent term features that preserve semantic relations between terms, which can also be beneficial to sensitivity classification. In this work, we present a thorough evaluation of the effectiveness of semantic word embedding features, along with term and grammatical features, for sensitivity classification. On a test collection of government documents containing real sensitivities, we show that extending text classification with semantic features and additional term n-grams results in significant improvements in classification effectiveness, correctly classifying 9.99% more sensitive documents compared to the text classification baseline.

Section: Related Workmentioning

confidence: 99%

Enhancing Sensitivity Classification with Semantic Features Using Word Embeddings

McDonald

Lecture Notes in Computer Science

Ounis

2017

“…The U-Theoretic approach described in [2] tackles SATC in a "multi-label multi-class" context, i.e., there is a set of classes C = {c1, . .…”

Section: Satc Sensitivity Identificationmentioning

confidence: 99%

“…SATC (see [2,5,6]) is defined as the task of ranking a set D of automatically classified textual documents in such a way that, if a human annotator validates the documents in a top-ranked portion of D with the goal of increasing the overall classification accuracy of D, the expected increase in accuracy is maximized. Therefore, we envisage our annotators as validating documents by sensitivity, starting from the top of the ranked list we generate, and working downwards (until they are confident that the dataset has been cleared up, or until the budget for annotation work has been spent).…”

Section: Introductionmentioning

confidence: 99%

“…We approach SATC by adopting the utility-theoretic approach of [2] (hereafter: U-Theoretic). Essentially, U-Theoretic ranks the automatically labelled documents by taking two factors into consideration, i.e., 1. the probability that the document has been misclassified by the classifier (documents with high probability of misclassification should be ranked higher, since no benefit will occur from validating a correctly classified document), and 2. the increase (or gain) in the overall accuracy of the automatically labelled set that occurs if the document is validated (documents that bring about a higher gain should be ranked higher).…”

Section: Introductionmentioning

confidence: 99%

“…Concerning (2), while the same gain occurs from validating either a true positive (TP) or a true negative (TN) -this gain is 0, since the human annotator will not change their labels -the gain that occurs from validating a false positive (FP) or a false negative (FN) may be, as shown in [2], different. When the two gains are different, the utility-theoretic approach tested in this paper:…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Semi-Automated Text Classification for Sensitivity Identification

Berardi

Esuli

Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

et al. 2015

Self Cite

Sensitive documents are those that cannot be made public, e.g., for personal or organizational privacy reasons. For instance, documents requested through Freedom of Information mechanisms must be manually reviewed for the presence of sensitive information before their actual release. Hence, tools that can assist human reviewers in spotting sensitive information are of great value to government organizations subject to Freedom of Information laws. We look at sensitivity identification in terms of semi-automated text classification (SATC), the task of ranking automatically classified documents so as to optimize the cost-effectiveness of human post-checking work. We use a recently proposed utilitytheoretic approach to SATC that explicitly optimizes the chosen effectiveness function when ranking the documents by sensitivity; this is especially useful in our case, since sensitivity identification is a recall-oriented task, thus requiring the use of a recall-oriented evaluation measure such as F2. We show the validity of this approach by running experiments on a multi-label multi-class dataset of government documents manually annotated according to different types of sensitivity.

Active Learning Strategies for Technology Assisted Sensitivity Review

McDonald

Lecture Notes in Computer Science

Ounis

2018

Abstract. Government documents must be reviewed to identify and protect any sensitive information, such as personal information, before the documents can be released to the public. However, in the era of digital government documents, such as e-mail, traditional sensitivity review procedures are no longer practical, for example due to the volume of documents to be reviewed. Therefore, there is a need for new technology assisted review protocols to integrate automatic sensitivity classification into the sensitivity review process. Moreover, to effectively assist sensitivity review, such assistive technologies must incorporate reviewer feedback to enable sensitivity classifiers to quickly learn and adapt to the sensitivities within a collection, when the types of sensitivity are not known a priori. In this work, we present a thorough evaluation of active learning strategies for sensitivity review. Moreover, we present an active learning strategy that integrates reviewer feedback, from sensitive text annotations, to identify features of sensitivity that enable us to learn an effective sensitivity classifier (0.7 Balanced Accuracy) using significantly less reviewer effort, according to the sign test (p < 0.01). Moreover, this approach results in a 51% reduction in the number of documents required to be reviewed to achieve the same level of classification accuracy, compared to when the approach is deployed without annotation features.