Graph-Homomorphic Perturbations for Private Decentralized Learning

Abstract

Decentralized algorithms for stochastic optimization and learning rely on the diffusion of information through repeated local exchanges of intermediate estimates. Such structures are particularly appealing in situations where agents may be hesitant to share raw data due to privacy concerns. Nevertheless, in the absence of additional privacy-preserving mechanisms, the exchange of local estimates, which are generated based on private data can allow for the inference of the data itself. The most common mechanism for guaranteeing privacy is the addition of perturbations to local estimates before broadcasting. These perturbations are generally chosen independently at every agent, resulting in a significant performance loss. We propose an alternative scheme, which constructs perturbations according to a particular nullspace condition, allowing them to be invisible (to first order in the step-size) to the network centroid, while preserving privacy guarantees. The analysis allows for general nonconvex loss functions, and is hence applicable to a large number of machine learning and signal processing problems, including deep learning.

Privatized Graph Federated Learning

“…In particular, we introduce the graph federated architec-ture, which consists of multiple servers, and we privatize the algorithm by ensuring the communication ocuring between the servers and the clients is secure. Graph homomorphic perturbations, which were initially introduced in [6], focus on the communication between servers. They are based on adding correlated noise to the messages sent between servers such that the noise cancels out if we were to take the average of all messages across all servers.…”

Privatized Graph Federated Learning

“…Contrarily, the agent-server noise components' subscripts are separated by a comma to highlight a hierarchical structure. Thus, the privatized algorithm can be written as a client update step (6), a server aggregation step (7), and a server combination step (8):…”

“…To achieve this, we introduce graph homomorphic perturbations [7] defined as follows. We assume each server p draws a sample g p,i independently from the Laplace distribution Lap(0, σ g / √ 2) with variance σ 2 g .…”

“…we introduce the graph federated architecture, which consists of multiple servers, and we privatize the algorithm by ensuring the communication occuring between the servers and the clients is secure. Graph homomorphic perturbationz, which were initially introduced in [7], focus on the communication between servers. They are based on adding correlated noise to the messages sent between servers such that the noise cancels out if we were to take the average of all messages across all servers.…”

Enforcing Privacy in Distributed Learning with Performance Guarantees

“…On the other hand, differentially private methods mask the messages by adding some random noise [36]- [46]. They are simple to implement, but they introduce errors into the learned model and reduce the overall utility of the network.…”