Fair Inputs and Fair Outputs: The Incompatibility of Fairness in Privacy and Accuracy

Rastegarpanah, Bashir; Crovella, Mark; Gummadi, Krishna P.

doi:10.1145/3386392.3399568

Cited by 4 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several works are concerned with implementing (Goldsteen et al 2021) or auditing compliance with this principle (Rastegarpanah, Gummadi, and Crovella 2021). Rastegarpanah et al (Rastegarpanah, Crovella, and Gummadi 2020) consider decision systems that can handle optional features from a data minimization perspective where the decision maker decides which features are collected for each individual. This principle is distinct from the "right to be forgotten" (Biega et al 2020), which enables individuals to submit requests to have their data deleted.…”

Section: Related Workmentioning

confidence: 99%

I Prefer Not to Say: Protecting User Consent in Models with Optional Personal Data

Leemann,

Pawelczyk,

Eberle

et al. 2024

AAAI

View full text Add to dashboard Cite

We examine machine learning models in a setup where individuals have the choice to share optional personal information with a decision-making system, as seen in modern insurance pricing models. Some users consent to their data being used whereas others object and keep their data undisclosed. In this work, we show that the decision not to share data can be considered as information in itself that should be protected to respect users' privacy. This observation raises the overlooked problem of how to ensure that users who protect their personal data do not suffer any disadvantages as a result. To address this problem, we formalize protection requirements for models which only use the information for which active user consent was obtained. This excludes implicit information contained in the decision to share data or not. We offer the first solution to this problem by proposing the notion of Protected User Consent (PUC), which we prove to be loss-optimal under our protection requirement. We observe that privacy and performance are not fundamentally at odds with each other and that it is possible for a decision maker to benefit from additional data while respecting users' consent. To learn PUC-compliant models, we devise a model-agnostic data augmentation strategy with finite sample convergence guarantees. Finally, we analyze the implications of PUC on challenging real datasets, tasks, and models.

show abstract

Section: Related Workmentioning

confidence: 99%

I Prefer Not to Say: Protecting User Consent in Models with Optional Personal Data

Leemann,

Pawelczyk,

Eberle

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Additionally, fair models trained with the constraint of group fairness are more vulnerable to membership inference attacks [6]. The tradeoffs and (in)compatibility of fairness, privacy and accuracy have been theoretically studied [9,37].…”

Section: Related Workmentioning

confidence: 99%

Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attacks

Aalmoes¹,

Duddu²,

Boutet³

2022

Preprint

View full text Add to dashboard Cite

Machine learning (ML) models have been deployed for high-stakes applications (e.g., criminal justice system). Due to class imbalance in the sensitive attribute observed in the datasets, ML models are unfair on minority subgroups identified by a sensitive attribute, such as Race and Sex. Fairness algorithms, specially in-processing algorithms, ensure model predictions are independent of sensitive attribute for fair classification across different subgroups (e.g., male and female; white and non-white). Furthermore, ML models are vulnerable to attribute inference attacks where an adversary can identify the values of sensitive attribute by exploiting their distinguishable model predictions. Despite privacy and fairness being important pillars of trustworthy ML, the privacy risk introduced by fairness algorithms with respect to attribute leakage has not been studied. In addition to different fairness metrics, we identify attribute inference attacks as an effective measure for auditing blackbox fairness algorithms to enable model builder to account for privacy and fairness in the model design. More precisely, we proposed Dikaios, a privacy auditing tool for fairness algorithms which leveraged a new effective attribute inference attack that account for the class imbalance in sensitive attributes through an adaptive prediction threshold. Dikaios can be used by model builders to estimate the attribute privacy risks of their model with or without the sensitive attribute in model training. We exhaustively evaluated Dikaios to perform a privacy audit of two in-processing group fairness algorithms (i.e., reductions and adversarial debiasing) over five datasets. First, we show that our attribute inference attack with adaptive prediction threshold significantly outperforms prior attacks, and second, we highlighted the limitations of in-processing fairness algorithms to ensure indistinguishable predictions across different values of sensitive attributes. Indeed, the attribute privacy risk of these in-processing fairness schemes is highly variable according to the proportion of the sensitive attributes in the dataset. This unpredictable effect of fairness mechanisms on the attribute privacy risk can be an important limitation on their utilization which has to be accounted by the model builder.

show abstract

“…Rastegarpanah, Crovella and Gummadi [3] looked into the fairness notions for algorithmic decision-making systems. The authors demonstrate that these systems should expand to incorporate the inputs used by a system.…”

Section: Accepted Papersmentioning

confidence: 99%

FairUMAP 2020: The 3rd Workshop on Fairness in User Modeling, Adaptation and Personalization

Mobasher

Kleanthous

Ekstrand

et al. 2020

Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization

View full text Add to dashboard Cite

show abstract

Fair Inputs and Fair Outputs: The Incompatibility of Fairness in Privacy and Accuracy

Cited by 4 publications

References 34 publications

I Prefer Not to Say: Protecting User Consent in Models with Optional Personal Data

I Prefer Not to Say: Protecting User Consent in Models with Optional Personal Data

Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attacks

FairUMAP 2020: The 3rd Workshop on Fairness in User Modeling, Adaptation and Personalization

Contact Info

Product

Resources

About