No abstract

Federated Learning enables mobile devices to collaboratively learn a shared inference model while keeping all the training data on a user's device, decoupling the ability to do machine learning from the need to store the data in the cloud. Existing work on federated learning with limited communication demonstrates how random rotation can enable users' model updates to be quantized much more efficiently, reducing the communication cost between users and the server. Meanwhile, secure aggregation enables the server to learn an aggregate of at least a threshold number of device's model contributions without observing any individual device's contribution in unaggregated form. In this paper, we highlight some of the challenges of setting the parameters for secure aggregation to achieve communication efficiency, especially in the context of the aggressively quantized inputs enabled by random rotation. We then develop a recipe for auto-tuning communication-efficient secure aggregation, based on specific properties of random rotation and secure aggregation -namely, the predictable distribution of vector entries postrotation and the modular wrapping inherent in secure aggregation. We present both theoretical results and initial experiments.

We consider training models with differential privacy (DP) using mini-batch gradients. The existing state-ofthe-art, Differentially Private Stochastic Gradient Descent (DP-SGD), requires privacy amplification by sampling or shuffling to obtain the best privacy/accuracy/computation trade-offs. Unfortunately, the precise requirements on exact sampling and shuffling can be hard to obtain in important practical scenarios, particularly federated learning (FL). We design and analyze a DP variant of Follow-The-Regularized-Leader (DP-FTRL) that compares favorably (both theoretically and empirically) to amplified DP-SGD, while allowing for much more flexible data access patterns. DP-FTRL does not use any form of privacy amplification. * Google. {kairouz, mcmahan, shuangsong, omthkkr, athakurta, xuzheng}@google.com1 Gradient computed on a subset of the training examples, also called a mini-batch.2 One can also create a mini-batch with Poisson sampling [1,49,73], except the batch size is now a random variable. For brevity, we focus on the fixed batch setting.3 A similar argument holds for amplification by shuffling [24,30], when the data are uniformly shuffled at the beginning of every epoch.We do not consider privacy amplification by iteration [28] in this paper, as it only applies to smooth convex functions.

Building privacy-preserving systems for machine learning and data science on decentralized data.

Motivated by differentially-private (DP) training of machine learning models and other applications, we investigate the problem of computing prefix sums in the online (streaming) setting with DP. This problem has previously been addressed by special-purpose tree aggregation schemes with hand-crafted estimators. We show that these previous schemes can all be viewed as specific instances of a broad class of matrix-factorization-based DP mechanisms, and that in fact much better mechanisms exist in this class.In particular, we characterize optimal factorizations of linear queries under online constraints, deriving existence, uniqueness, and explicit expressions that allow us to efficiently compute optimal mechanisms, including for online prefix sums. These solutions improve over the existing state-ofthe-art by a significant constant factor, and avoid some of the artifacts introduced by the use of the tree data structure.