Wyner-Ziv Estimators: Efficient Distributed Mean Estimation with Side Information

Abstract

Communication efficient distributed mean estimation is an important primitive that arises in many distributed learning and optimization scenarios such as federated learning. Without any probabilistic assumptions on the underlying data, we study the problem of distributed mean estimation where the server has access to side information. We propose Wyner-Ziv estimators, which are communication and computationally efficient and near-optimal when an upper bound for the distance between the side information and the data is known. As a corollary, we also show that our algorithms provide efficient schemes for the classic Wyner-Ziv problem in information theory. In a different direction, when there is no knowledge assumed about the distance between side information and the data, we present an alternative Wyner-Ziv estimator that uses correlated sampling. This latter setting offers universal recovery guarantees, and perhaps will be of interest in practice when the number of users is large and keeping track of the distances between the data and the side information may not be possible.

Improved Communication Efficiency for Distributed Mean Estimation with Side Information

“…, x n ) in the previous iterations. We focus on non-interactive protocols and study the r-bit simultaneous message passing (SMP) protocol similar to that in [16]. The r-bit SMP protocol π = (π 1 , .…”

“…To alleviate the communication bottleneck, gradient compression [2]- [8] and efficient mean estimator [9]- [15] have been investigated to reduce the communication load. Recently, [16] studied distributed mean estimation with side information at the server, and proposed Wyner-Ziv estimators that require no probabilistic assumption on the clients data.…”

“…The motivation is based on the fact that the server could store publicly accessible data, and also at each iteration, the server has already received data sent by clients at previous iterations, which can be viewed as side information. Rather than using random coding with joint typicality tools such as in [17], [20], which is impractical to implement, we follow the work in [16] which proposed a Wyner-Ziv estimator based on coset coding. Unfortunately, they only utilized the side information at the server, but failed to exploit correlation between clients' vectors.…”

“…Inspired by Wyner-Ziv and Slepian-Wolf coding, we propose a practical scheme based on the coset coding and jointly exploit the side information at the server and correlation between clients' data. Note that in our scheme we must address an ambiguity problem not existing in [16] or in the classic Wyner-Ziv coding. In more detail, since each client compresses its data and sends it to the server, the server only observes a lossy version of clients' vectors.…”

“…We summarize our contributions as follows: 1) We propose a new estimator that improves the estimator in [16] by jointly exploiting the side information at the server and the correlation between clients' data; 2) We derive an upper bound of estimation error of the proposed estimator; 3) We provide two greedy algorithms on how to choose input parameters for the estimator, and characterize the parameter regions in which our estimator has a tighter upper bound than of the previous estimator.…”