Private Read Update Write (PRUW) with Storage Constrained Databases

Abstract

We investigate the problem of private read update write (PRUW) with heterogeneous storage constrained databases in federated submodel learning (FSL). In FSL a machine learning (ML) model is divided into multiple submodels based on different types of data used to train it. A given user downloads, updates and uploads the updates back to a single submodel of interest, based on the type of user's local data. With PRUW, the process of reading (downloading) and writing (uploading) is carried out such that information theoretic privacy of the updating submodel index and the values of updates is guaranteed. We consider the practical scenario where the submodels are stored in databases with arbitrary (heterogeneous) storage constraints, and provide a PRUW scheme with a storage mechanism that utilizes submodel partitioning and encoding to minimize the communication cost.

Private Federated Submodel Learning with Sparsification

“…Note from (35) that for segment 1, the two real sparse subpackets 2 and 4 have been correctly updated, while ensuring that the rest of the subpackets remain the same, without revealing the real subpacket indices 2 and 4 to any of the databases. The resulting writing cost is given by,…”

“…Apart from the privacy leakage, another drawback of FL is the large communication cost incurred by sharing model parameters and updates with millions of users in multiple rounds. Some of the solutions to this problem include, gradient quantization [25][26][27][28], federated submodel learning (FSL) [29][30][31][32][33][34][35][36][37], and gradient sparsification [38][39][40][41][42][43][44][45]. In gradient quantization, the values of the gradients are quantized and represented with a fewer number of bits.…”

“…This results in a significantly large communication cost, which is one of the main drawbacks of FL. Many solutions have been proposed to reduce the communication overhead in FL, such as gradient sparsification [3][4][5][6][7][8], where the users only download and upload a selected fraction of parameters and updates, gradient quantization [9][10][11][12] in which the updates are quantized to be represented using a fewer number of bits, and federated submodel learning (FSL) [13][14][15][16][17][18][19][20], where the ML model is divided into multiple submodels based on different types of data used for training. In FSL, a given user only downloads and updates the submodel relevant to the user's local data, which significantly reduces the communication cost.…”