Semi-Automated Text Classification for Sensitivity Identification

Abstract

Sensitive documents are those that cannot be made public, e.g., for personal or organizational privacy reasons. For instance, documents requested through Freedom of Information mechanisms must be manually reviewed for the presence of sensitive information before their actual release. Hence, tools that can assist human reviewers in spotting sensitive information are of great value to government organizations subject to Freedom of Information laws. We look at sensitivity identification in terms of semi-automated text classification (SATC), the task of ranking automatically classified documents so as to optimize the cost-effectiveness of human post-checking work. We use a recently proposed utilitytheoretic approach to SATC that explicitly optimizes the chosen effectiveness function when ranking the documents by sensitivity; this is especially useful in our case, since sensitivity identification is a recall-oriented task, thus requiring the use of a recall-oriented evaluation measure such as F2. We show the validity of this approach by running experiments on a multi-label multi-class dataset of government documents manually annotated according to different types of sensitivity.

Enhancing Sensitivity Classification with Semantic Features Using Word Embeddings

“…As our classifier, we use SVM with a linear kernel and C = 1.0, since this theoretically motivated, default, parameter setting has been shown to provide the best effectiveness for text classification [2,24]. We select F 2 as our main metric since sensitivity classification is a recall oriented task [3,4], where the consequences of missclassifying a sensitive document are much greater than miss-classifying a not-sensitive document. We also report the standard F-Measure (F 1 ) and, to account for class imbalance, we report Balanced Accuracy (BAC), where 0.5 BAC is random.…”

“…Berardi et al [4] built on the work of McDonald et al [3] to optimise the costeffectiveness of sensitivity reviewers. In that work, Berardi et al deployed a utilitytheoretic ranking approach for semi-automatic text classification [8].…”

“…Although sensitivity classification is not a topic-oriented task [4], text classification has been shown to be a strong baseline approach [3,4]. A popular, and effective, extension to text classification is to include additional n-gram term features [2].…”

“…Text classification [2] is one approach that has been shown to be promising as a basis for automatic sensitivity identification algorithms [3,4]. Usually, a text classification model is learned by observing statistical patterns in the distributions of individual key terms from example documents.…”

“…Usually, a text classification model is learned by observing statistical patterns in the distributions of individual key terms from example documents. However, the potential effectiveness of sensitivity classification from single-term observations is limited, due to the fact that sensitivity classification is not a topic-oriented task [4] and, moreover, sensitivity tends to arise as a product of specific factors. For example, International Relations sensitivities are often a product of who said what about whom.…”