Distributed Differentially Private Mutual Information Ranking and Its Applications

Abstract

Computation of Mutual Information (MI) helps understand the amount of information shared between a pair of random variables. Automated feature selection techniques based on MI ranking are regularly used to extract information from sensitive datasets exceeding petabytes in size, over millions of features and classes. Series of one-vs-all MI computations can be cascaded to produce n-fold MI results, rapidly pinpointing informative relationships. This ability to quickly pinpoint the most informative relationships from datasets of billions of users creates privacy concerns. In this paper, we present Distributed Differentially Private Mutual Information (DDP-MI), a privacysafe fast batch MI, across various scenarios such as feature selection, segmentation, ranking, and query expansion. This distributed implementation is protected with global model differential privacy to provide strong assurances against a wide range of privacy attacks. We also show that our DDP-MI can substantially improve the efficiency of MI calculations compared to standard implementations on a large-scale public dataset.

Differentially Private Feature Selection Based on Dynamic Relevance for Correlated Data

“…It returned differentially private data aggregates from a low-dimensional dataset, and a two-stage noise injection was used to satisfy the trade-off between privacy and utility of data aggregates. To solve the privacy problem caused by automatic selection techniques based on MI ranking, Srivastava et al [11] proposed a Distributed Differentially Private Mutual Information (DDP-MI), as a privacy-safe batch MI, and was used in some scenarios such as feature selection, segmentation, ranking and query expansion. Moreover, the distributed implementation provided a strong guarantee against various privacy attacks and substantially improved the efficiency of MI calculations.…”

“…where the first inequality is from triangle inequality, and the second inequality is from Equation (11). Therefore, the feature selection procedure with differentially private F-score satisfies ε 1 -differential privacy.…”