Sample-efficient proper PAC learning with approximate differential privacy

Abstract

In this paper we prove that the sample complexity of properly learning a class of Littlestone dimension d with approximate differential privacy is Õpd 6 q, ignoring privacy and accuracy parameters. This result answers a question of Bun et al. (FOCS 2020) by improving upon their upper bound of 2 Opdq on the sample complexity. Prior to our work, finiteness of the sample complexity for privately learning a class of finite Littlestone dimension was only known for improper private learners, and the fact that our learner is proper answers another question of Bun et al., which was also asked by Bousquet et al. (NeurIPS 2020). Using machinery developed by Bousquet et al., we then show that the sample complexity of sanitizing a binary hypothesis class is at most polynomial in its Littlestone dimension and dual Littlestone dimension. This implies that a class is sanitizable if and only if it has finite Littlestone dimension. An important ingredient of our proofs is a new property of binary hypothesis classes that we call irreducibility, which may be of independent interest.

Littlestone Classes are Privately Online Learnable

“…We obtain, though, a strong deterioration in terms of the Littlestone dimension -sublinear dependece vs. double exponential dependence. As discussed, Ghazi et al [20] improved the dependence in the batch case to polynomial, and it remains an open question if similar improvement is applicable in the online case. We next turn to the adversarial case Theorem 4.2 (Private Adaptive online-learning).…”

“…In addition to [11,20] which establish private learning algorithms for classes with finite Littlestone dimension in the i.i.d. (offline) setting, there has been an extensive line of work on private learning algorithms in the offline setting: [29,7,5,19] study the complexity of private learning with pure differential privacy, [26,9,10,4] study the sample complexity of privately learning thresholds, and [27,28,6] study the sample complexity of privately learning halfspaces.…”

“…The notion of differential privacy has emerged as a central tool which can be used to formally reason about the privacy-accuracy tradeoffs one must make in the process of analyzing and learning from data. A considerable body of literature on differentially private machine learning has resulted, ranging from empirical works which train deep neural networks with a differentially private form of stochastic gradient descent [1], to a recent line of theoretical works which aim to characterize the optimal sample complexity of privately learning an arbitrary hypothesis class [3,11,20].…”

“…Recently it was shown by Alon et al [3] and Bun et al [11] (later to be improved by Ghazi et al [20]) that finiteness of the Littlestone dimension is necessary and sufficient for private learnability in the offline setting, namely with i.i.d. data (and both in the realizable and agnostic settings).…”

Private learning implies quantum stability

“…2. In a very recent work, Ghazi et al [GGKM20] improved upon the result of Bun et al [BLM20] by showing that a polynomial blow-up in sample complexity suffices in going from online learning to differential privacy, which is exponentially better than the result of Bun et al [BLM20].…”