Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attacks

Abstract

Machine learning (ML) models have been deployed for high-stakes applications (e.g., criminal justice system). Due to class imbalance in the sensitive attribute observed in the datasets, ML models are unfair on minority subgroups identified by a sensitive attribute, such as Race and Sex. Fairness algorithms, specially in-processing algorithms, ensure model predictions are independent of sensitive attribute for fair classification across different subgroups (e.g., male and female; white and non-white). Furthermore, ML models are vulnerable to attribute inference attacks where an adversary can identify the values of sensitive attribute by exploiting their distinguishable model predictions. Despite privacy and fairness being important pillars of trustworthy ML, the privacy risk introduced by fairness algorithms with respect to attribute leakage has not been studied. In addition to different fairness metrics, we identify attribute inference attacks as an effective measure for auditing blackbox fairness algorithms to enable model builder to account for privacy and fairness in the model design. More precisely, we proposed Dikaios, a privacy auditing tool for fairness algorithms which leveraged a new effective attribute inference attack that account for the class imbalance in sensitive attributes through an adaptive prediction threshold. Dikaios can be used by model builders to estimate the attribute privacy risks of their model with or without the sensitive attribute in model training. We exhaustively evaluated Dikaios to perform a privacy audit of two in-processing group fairness algorithms (i.e., reductions and adversarial debiasing) over five datasets. First, we show that our attribute inference attack with adaptive prediction threshold significantly outperforms prior attacks, and second, we highlighted the limitations of in-processing fairness algorithms to ensure indistinguishable predictions across different values of sensitive attributes. Indeed, the attribute privacy risk of these in-processing fairness schemes is highly variable according to the proportion of the sensitive attributes in the dataset. This unpredictable effect of fairness mechanisms on the attribute privacy risk can be an important limitation on their utilization which has to be accounted by the model builder.