Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction

Acharya, Jayadev; Canonne, Clément L.; Tyagi, Himanshu

doi:10.1109/tit.2020.3028440

Cited by 63 publications

(187 citation statements)

References 31 publications

Supporting

Mentioning

183

Contrasting

Order By: Relevance

“…Total population size 6 -10 10 devices Devices selected for one round of training 50 -5000 Total devices that participate in training one model 10 5 -10 7 Number of rounds for model convergence 500 -10000 Wall-clock training time 1 -10 days Table 2: Order-of-magnitude sizes for typical cross-device federated learning applications.…”

Section: Simulation Prototyping (Optional)mentioning

confidence: 99%

“…However, it remains unclear if communication cost can be further reduced, and whether any of these methods or their combinations can come close to providing optimal trade-offs between communication and accuracy in federated learning. Characterizing such fundamental trade-offs between accuracy and communication has been of recent interest in theoretical statistics [507,89,221,7,49,444,50]. These works characterize the optimal minimax rates for distributed statistical estimation and learning under communication constraints.…”

Section: Communication and Compressionmentioning

confidence: 99%

“…Among them, (a) has the potential for the most significant practical impact in terms of total runtime. This is both because clients' connections generally have slower upload than download bandwidth 7 -and thus there is more to be gained, compared to (b) -and because the effects of averaging across many clients can enable more aggressive lossy compression schemes. Usually, (c) could be realized jointly with (a) and (b) by specific methods.…”

Section: Communication and Compressionmentioning

confidence: 99%

“…In this section, we use "gradient compression" to include compression applied to any model update, such as the updates produced by Federated Averaging when clients take multiple gradient steps 7. See for instance https://www.speedtest.net/reports/…”

mentioning

confidence: 99%

See 3 more Smart Citations

Advances and Open Problems in Federated Learning

Kairouz

McMahan²

2021

FNT in Machine Learning

2,473

1,214

View full text Add to dashboard Cite

Federated learning (FL) is a machine learning setting where many clients (e.g. mobile devices or whole organizations) collaboratively train a model under the orchestration of a central server (e.g. service provider), while keeping the training data decentralized. FL embodies the principles of focused data collection and minimization, and can mitigate many of the systemic privacy risks and costs resulting from traditional, centralized machine learning and data science approaches. Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.

show abstract

Section: Simulation Prototyping (Optional)mentioning

confidence: 99%

Section: Communication and Compressionmentioning

confidence: 99%

Section: Communication and Compressionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Advances and Open Problems in Federated Learning

Kairouz

McMahan²

2021

FNT in Machine Learning

2,473

1,214

View full text Add to dashboard Cite

show abstract

“…Identical and closely related models are often studied in the context of distributed statistical estimation with communication constraints (e.g. (Luo, 2005;Rajagopal et al, 2006;Ribeiro and Giannakis, 2006;Zhang et al, 2013;Steinhardt and Duchi, 2015;Suresh et al, 2017;Acharya et al, 2020aAcharya et al, , 2019bAcharya et al, ,a, 2020b). As in the setting of LDP, the number of rounds of interaction that the server uses to solve a learning problem is a critical resource.…”

Section: Introductionmentioning

confidence: 99%

Interaction is Necessary for Distributed Learning with Privacy or Communication Constraints

Dagan

Feldman

2021

JPC

View full text Add to dashboard Cite

Local differential privacy (LDP) is a model where users send privatized data to an untrusted central server whose goal it to solve some data analysis task. In the non-interactive version of this model the protocol consists of a single round in which a server sends requests to all users then receives their responses. This version is deployed in industry due to its practical advantages and has attracted significant research interest. Our main result is an exponential lower bound on the number of samples necessary to solve the standard task of learning a large-margin linear separator in the non-interactive LDP model. Via a standard reduction this lower bound implies an exponential lower bound for stochastic convex optimization and specifically, for learning linear models with a convex, Lipschitz and smooth loss. These results answer the questions posed by Smith, Thakurta, and Upadhyay (IEEE Symposium on Security and Privacy 2017) and Daniely and Feldman (NeurIPS 2019). Our lower bound relies on a new technique for constructing pairs of distributions with nearly matching moments but whose supports can be nearly separated by a large margin hyperplane. These lower bounds also hold in the model where communication from each user is limited and follow from a lower bound on learning using non-adaptive statistical queries.

show abstract

BAFFLE: A Baseline of Backpropagation-Free Federated Learning

Feng,

Pang,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Inference Under Information Constraints I: Lower Bounds From Chi-Square Contraction

Cited by 63 publications

References 31 publications

Advances and Open Problems in Federated Learning

Advances and Open Problems in Federated Learning

Interaction is Necessary for Distributed Learning with Privacy or Communication Constraints

BAFFLE: A Baseline of Backpropagation-Free Federated Learning

Contact Info

Product

Resources

About