Analyzing Information Leakage of Updates to Natural Language Models

Zanella-Béguelin, Santiago; Wutschitz, Lukas; Tople, Shruti; Rühle, Victor; Paverd, Andrew; Ohrimenko, Olga; Köpf, Boris; Brockschmidt, Marc

doi:10.1145/3372297.3417880

Cited by 61 publications

(63 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Training data extraction refers to the risk of partially extracting training samples by interacting with a trained language model [80,12,99,13]. An adversary can use use membership inference attacks as an oracle to generate sentence samples that have a high chance to be in the training set.…”

Section: Training Data Extractionmentioning

confidence: 99%

What Does it Mean for a Language Model to Preserve Privacy?

Brown¹,

Lee²,

Mireshghallah³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus, there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.

show abstract

Section: Training Data Extractionmentioning

confidence: 99%

What Does it Mean for a Language Model to Preserve Privacy?

Brown¹,

Lee²,

Mireshghallah³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For instance, several approaches have been proposed in recent years for 'machine unlearning', allowing to erase data from already trained models [19,9,12,76]. However, recent results have also shown that: 1) it is possible to reveal details from an initial dataset even when a model was subsequently retrained on a redacted version [85];…”

Section: Technical Issuesmentioning

confidence: 99%

A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication

Luccioni,

Corry,

Sridharan

et al. 2021

Preprint

View full text Add to dashboard Cite

What happens when a machine learning dataset is deprecated for legal, ethical, or technical reasons, but continues to be widely used? In this paper, we examine the public afterlives of several prominent deprecated or redacted datasets, including ImageNet, 80 Million Tiny Images, MS-Celeb-1M, Duke MTMC, Brainwash, and HRT Transgender, in order to inform a framework for more consistent, ethical, and accountable dataset deprecation. Building on prior research [27,35,61], we find that there is a lack of consistency, transparency, and centralized sourcing of information on the deprecation of datasets, and as such, these datasets and their derivatives continue to be cited in papers and circulate online. These datasets that never die -which we term "zombie datasets" -continue to inform the design of production-level systems, causing technical, legal, and ethical challenges; in so doing, they risk perpetuating the harms that prompted their supposed withdrawal, including concerns around bias, discrimination, and privacy. Based on this analysis, we propose a Dataset Deprecation Framework that includes considerations of risk, mitigation of impact, appeal mechanisms, timeline, post-deprecation protocol, and publication checks that can be adapted and implemented by the machine learning community. Drawing on work on datasheets and checklists [29,48], we further offer two sample dataset deprecation sheets and propose a centralized repository that tracks which datasets have been deprecated and could be incorporated into the publication protocols of venues like NeurIPS.

show abstract

“…However, in many cases, a recommender system also needs to forget certain sensitive data and its complete lineage, which is called Recommendation Unlearning in this paper. Consider privacy first, recent researches have shown that users' sensitive information could be leaked from the trained models, e.g., recommender systems [50], big pretrained [4] and finetuned natural language models [49]. In such cases, users desire a tool to erase the impacts of their sensitive information from the trained models.…”

Section: Introductionmentioning

confidence: 99%

Recommendation Unlearning

Chen¹,

Sun²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recommender systems provide essential web services by learning users' personal preferences from collected data. However, in many cases, systems also need to forget some training data. From the perspective of privacy, users desire a tool to erase the impacts of their sensitive data from the trained models. From the perspective of utility, if a system's utility is damaged by some bad data, the system needs to forget such data to regain utility. While unlearning is very important, it has not been well-considered in existing recommender systems. Although there are some researches have studied the problem of machine unlearning, existing methods can not be directly applied to recommendation as they are unable to consider the collaborative information.In this paper, we propose RecEraser, a general and efficient machine unlearning framework tailored to recommendation tasks. The main idea of RecEraser is to divide the training set into multiple shards and train submodels with these shards. Specifically, to keep the collaborative information of the data, we first design three novel data partition algorithms to divide training data into balanced groups. We then further propose an adaptive aggregation method to improve the global model utility. Experimental results on three public benchmarks show that RecEraser can not only achieve efficient unlearning but also outperform the state-of-the-art unlearning methods in terms of model utility. The source code can be found at https://github.com/chenchongthu/Recommendation-Unlearning CCS CONCEPTS• Information systems → Recommender systems; • Security and privacy → Privacy protections.

show abstract

Analyzing Information Leakage of Updates to Natural Language Models

Cited by 61 publications

References 6 publications

What Does it Mean for a Language Model to Preserve Privacy?

What Does it Mean for a Language Model to Preserve Privacy?

A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication

Recommendation Unlearning

Contact Info

Product

Resources

About