Mining Privacy Goals from Privacy Policies Using Hybridized Task Recomposition

Abstract

Privacy policies describe high-level goals for corporate data practices; regulators require industries to make available conspicuous, accurate privacy policies to their customers. Consequently, software requirements must conform to those privacy policies. To help stakeholders extract privacy goals from policies, we introduce a semiautomated framework that combines crowdworker annotations, natural language typed dependency parses, and a reusable lexicon to improve goal-extraction coverage, precision, and recall. The framework evaluation consists of a five-policy corpus governing web and mobile information systems, yielding an average precision of 0.73 and recall of 0.83. The results show that no single framework element alone is sufficient to extract goals; however, the overall framework compensates for elemental limitations. Human annotators are highly adaptive at discovering annotations in new texts, but those annotations can be inconsistent and incomplete; dependency parsers lack sophisticated, tacit knowledge, but they can perform exhaustive text search for prospective requirements indicators; and while the lexicon may never completely saturate, the lexicon terms can be reliably used to improve recall. Lexical reuse reduces false negatives by 41%, increasing the average recall to 0.85. Last, crowd workers were able to identify and remove false positives by around 80%, which improves average precision to 0.93.

PrivOnto: A semantic framework for the analysis of privacy policies

“…Semantically-labeled privacy policies constitute an important resource for privacy analysts and regulators, but scaling the process of annotating natural language privacy policies accordingly can be challenging. As part of the efforts in the UPP project, we investigate the potential of crowdsourcing privacy policy analysis from non-experts, in combination with machine learning, in order to enable semi-or fully automated extraction of data practices and their attributes from privacy policy documents [3,5,48]. These efforts show promise for scaling up our analysis, which would enable further expansion of PrivOnto's knowledge base.…”

Çorba: crowdsourcing to obtain requirements from regulations and breaches

“…Breaux and Schaub report that they could reduce manual extraction cost by up to 60% for some policies while preserving task accuracy, and for some policies increase accuracy by 16%, based on their ways of task decomposition. They continue using crowdsourcing, combined with NLP techniques, to extract privacy goals [4]. Reidenberg et al [36] investigate how privacy policies are perceived by expert, knowledgeable, and typical users, and did not find significant differences among them.…”

Automated Extraction of Semantic Legal Metadata using Natural Language Processing

“…In relation to legal requirements specifically, Bhatia et al [9], [39] and Evans et al [40] apply constituency and dependency parsing for analyzing privacy policies. These threads of work have provided us with useful inspiration.…”

“…Within the limits and according to the provisions stated in this article, the municipal authorities may, in whole or in part, temporarily or permanently, regulate or prohibit traffic on the public roads of the territory of the municipality, provided that these municipal regulations concern the traffic on the municipal roads as well as on the national roads situated inside the municipality's agglomerations. requirements [1], [7], [8], [9], and transitioning from legal texts to formal specifications [10] or models [3], [8], [11]. In this paper, we concern ourselves with semantic legal metadata.…”