Communication-Efficient Federated Learning via Quantized Compressed Sensing

Abstract

In this paper, we present a communication-efficient federated learning framework inspired by quantized compressed sensing. The presented framework consists of gradient compression for wireless devices and gradient reconstruction for a parameter server (PS). Our strategy for gradient compression is to sequentially perform block sparsification, dimensional reduction, and quantization. Thanks to gradient sparsification and quantization, our strategy can achieve a higher compression ratio than one-bit gradient compression. For accurate aggregation of the local gradients from the compressed signals at the PS, we put forth an approximate minimum mean square error (MMSE) approach for gradient reconstruction using the expectation-maximization generalized-approximate-message-passing (EM-GAMP) algorithm.Assuming Bernoulli Gaussian-mixture prior, this algorithm iteratively updates the posterior mean and variance of local gradients from the compressed signals. We also present a low-complexity approach for the gradient reconstruction. In this approach, we use the Bussgang theorem to aggregate local gradients from the compressed signals, then compute an approximate MMSE estimate of the aggregated gradient using the EM-GAMP algorithm. We also provide a convergence rate analysis of the presented framework.Using the MNIST dataset, we demonstrate that the presented framework achieves almost identical performance with the case that performs no compression, while significantly reducing communication overhead for federated learning.

FedVQCS: Federated Learning via Vector Quantized Compressed Sensing

“…To overcome the limitation of the quantization-only approach, communication-efficient FL via scalar quantized compressed sensing (SQCS) has been explored in [15]- [17]. A key motivation in this approach is the sparsity of the local model update, obtained either naturally or by applying sparsification [18]- [23].…”

“…A one-bit SQCS approach for communication-efficient FL was studied in [15] and [16], in which the entries of the local model update after dimensionality reduction were independently quantized by a one-bit scalar quantizer. This approach was extended to a multi-bit SQCS approach in [17] which allows greater flexibility regarding the number of scalar quantization bits. The multi-bit SQCS approach was shown to provide a better tradeoff between communication efficiency and FL performance when compared with the one-bit SQCS approach [17].…”

“…This approach was extended to a multi-bit SQCS approach in [17] which allows greater flexibility regarding the number of scalar quantization bits. The multi-bit SQCS approach was shown to provide a better tradeoff between communication efficiency and FL performance when compared with the one-bit SQCS approach [17]. However, the scalar quantizer used in the existing SQCS approaches has a higher quantization error compared to a vector quantizer.…”

“…k , and we refer to this vector as the b-th local block update of device k in round t. Each device compresses these B local block updates in parallel by applying a local block compression process denoted by a function BlkComp(•). The motivation behind this block-wise compression is to reduce the computational complexity for reconstructing the global model update at the PS, as considered in [17]. The compressed local block updates are encoded into digital bits and then transmitted to the PS through wireless links.…”

“…Therefore, FedVQCS achieves further compression of the local model updates compared with the frameworks in [12]- [14], without significantly sacrificing the learning accuracy. The existing FL framework in [17], which is our prior work, also leverages the CS-based dimensionality reduction, but considers only scalar quantization. In general, the scalar quantization is inferior to the vector quantization considered in FedVQCS in terms of the quantization error.…”