The Effect of Text Ambiguity on creating Policy Knowledge Graphs

Abstract

A growing number of web and cloud-based products and services rely on data sharing between consumers, service providers, and their subsidiaries and third parties. There is a growing concern around the security and privacy of data in such large-scale shared architectures. Most organizations have a human-written privacy policy that discloses all the ways that data is shared, stored, and used. The organizational privacy policies must also be compliant with government and administrative regulations. This raises a major challenge for providers as they try to launch new services. Thus they are moving towards a system of automatic policy maintenance and regulatory compliance. This requires extracting policy from text documents and representing it in a semi-structured, machine-processable framework. The most popular method to this end is extracting policy information into a Knowledge Graph (KG). There exists a significant body of work that converts text descriptions of regulations into policies expressed in languages such as OWL and XACML and is grounded in the control-based schema by using NLP approaches. In this paper, we show that the NLP-based approaches to extract knowledge from written policy documents and representing them in enforceable Knowledge Graphs fail when the text policies are ambiguous. Ambiguity can arise from lack of clarity, misuse of syntax, and/or the use of complex language. We describe a system to extract features from a policy document that affect its ambiguity and classify the documents based on the level of ambiguity present. We validate this approach using human annotators. We show that a large number of documents in a popular privacy policy corpus (OPP-115) are ambiguous. This affects the ability to automatically monitor privacy policies. We show that for policies that are more ambiguous according to our proposed measure, NLP-based text segment classifiers are less accurate.

Understanding Website Privacy Policies—A Longitudinal Analysis Using Natural Language Processing

“…This in turn limits not only the ability of human readers to precisely interpret their contents, but also machines' ability to "understand" them. Kotal et al [57] found that NLP-based text segment classifiers are less accurate for policies that are more ambiguous.…”

“…Kaur et al [11] and Srinath et al [12] analyzed the use of ambiguous words in a corpus of 2000 policies. Furthermore, Kotal et al [57] studied the ambiguity in the OPP-115 dataset and showed that ambiguity negatively affects the ability to automatically evaluate privacy policies. Srinath et al [12] reported on privacy policy length and the use of vague words in their PrivaSeer corpus of policies.…”

A GDPR Compliant Approach to Assign Risk Levels to Privacy Policies

“…Our decision was based on our previous results, which contain several comparison tables [26]. We also selected SVM as our primary classifier, since it has the best performance among all previous studies that used the OPP-115 dataset and traditional ML [16,[26][27][28].…”

PriveTAB

“…Privacy in data sharing has been heavily discussed in the past decade. Organizations are invested in finding secure, automated solutions to collecting and sharing data [25], [24]. However, organizations are still sceptical about sharing their data for use in research.…”