ER-AE: Differentially Private Text Generation for Authorship Anonymization

Abstract

Most of privacy protection studies for textual data focus on removing explicit sensitive identifiers. However, personal writing style, as a strong indicator of the authorship, is often neglected. Recent studies, such as SynTF, have shown promising results on privacy-preserving text mining. However, their anonymization algorithm can only output numeric term vectors which are difficult for the recipients to interpret. We propose a novel text generation model with a two-set exponential mechanism for authorship anonymization. By augmenting the semantic information through a REINFORCE training reward function, the model can generate differentially private text that has a close semantic and similar grammatical structure to the original text while removing personal traits of the writing style. It does not assume any conditioned labels or paralleled text data for training. We evaluate the performance of the proposed model on the reallife peer reviews dataset and the Yelp review dataset. The result suggests that our model outperforms the state-of-the-art on semantic preservation, authorship obfuscation, and stylometric transformation.

DP-VAE: Human-Readable Text Anonymization for Online Reviews with Differentially Private Variational Autoencoders

“…Section 2.1) is a notion of privacy based on randomness, i.e., any non-trivial DP mechanism must be non-deterministic. In the context of textual data, existing DP mechanisms typically operate on a word-by-word basis [6,[23][24][25]63] by randomly determining each word in the output sequence. The common ground of these methods is to impose a probability distribution, locally at each position, over the words in the output space (or similarly, over the embedding space) and draw a word (or corresponding embedding) from that distribution.…”

“…Weggenmann and Kerschbaum [63] and Fernandes et al [23] approach anonymization through applying DP mechanisms on document term frequency vectors, obtaining differentially private text representations that are, however, not human-readable. Other methods relying on DP mechanisms apply these on a word-level by perturbing words or their embeddings and thus exchanging individual tokens in a given text [6,24,25]. Deep Learning based approaches for authorship obfuscation include back-translation [37,64] as well as the incorporation of GANs as discriminators [55].…”

“…While many DP mechanisms are readily available for structured data such as numbers or vectors, it is not always easy to apply DP to unstructured data such as text, which comes in varying lengths and with different ways to express the same idea. Existing DP approaches for text work on a per-word level [6,[23][24][25]63], thus obfuscating the original input while keeping it statistically relevant overall. However, each of these approaches has one or more undesired drawbacks: the output is a vector or Bag-of-Words (BoW) representation and thus not human-readable, the produced text is incoherent due to the words being perturbed individually, the DP guarantees only apply to texts of the same length, and/or the privacy loss 𝜖 grows linearly with the length of the output.…”

The Limits of Word Level Differential Privacy

“…While not humanreadable, these vector representations can be shared for automated processing, such as topic or sentiment inference and machine learning. To generate human-readable text, Bo et al (2021) employ an encoder-decoder model similar to ours, but without paraphrasing, and sample output words using (a two-set variant of) the Exponential mechanism (McSherry and Talwar, 2007). Weggenmann et al (2022) propose a differentially private variation of the variational autoencoder and use it as a sequenceto-sequence architecture for text anonymization.…”

“…Previous work in the field of authorship obfuscation mainly focuses on two different tasks, namely learning anonymous textual vector representations for downstream tasks (Coavoux et al, 2018a;Weggenmann and Kerschbaum, 2018;Fernandes et al, 2019;Mosallanezhad et al, 2019;Beigi et al, 2019) and the development of mechanisms that transform the input sentence to remove properties revealing the author and thus output human-readable text. Works within the second category (Feyisetan et al, 2019(Feyisetan et al, , 2020Xu et al, 2020b;Bo et al, 2021) typically follow a common word level framework which is characterized by the differentially private individual perturbation of word embeddings and the subsequent sampling of new words that are close to the perturbed vectors in the embedding space. Also, the majority of recent work proposing new methods for authorship obfuscation deals with the optimization and calibration of noise sampling mechanisms (Xu et al, 2020a) or the definition of new distributions to sample noise from (Feyisetan et al, 2019) as opposed to the development of entirely new methods.…”