Model Fusion with Kullback--Leibler Divergence

Yurochkin, Mikhail; Ghosh, Soumya K.; Solomon, Justin

doi:10.48550/arxiv.2007.06168

Cited by 3 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, their approach only makes a point estimate of local posterior and precludes using full information like covariances. Inspired by the model fusion work in [2], we extend the original PFNM by incorporating global posterior information during the matching progress. The proposed new method is theoretically proved to inherit non-parametric nature and shown to have better performance compared to PFNM through experiments.…”

Section: Related Workmentioning

confidence: 99%

“…, J s } and thus J −s ≤ J ≤ J −s + J s . The key step of whole matching process is the derivation of assignment cost specifications {C s ij } i,j (2). The overall matched aggregation procedure is summarized in Figure 1.…”

Section: Adaptive Matched Aggregation Formulationmentioning

confidence: 99%

“…That means all information applied for inferring is between a pair of neural components. Inspired by work in [2], it can also pose the problem of recovering a global model as that of minimizing a divergence to the local models. Specifically, by minimizing the following KL divergence KL(•||•) between the products of the global and local neural components:…”

Section: Gpi-fnmmentioning

confidence: 99%

“…Recently, [2] proposes a method to fuse a model's posterior distribution learned from heterogeneous datasets. The authors pose the problem of recovering a global distribution as that of minimizing the sum of Kullback-Leibler (KL) divergence from the local distributions while preserving the inherent permutation invariance of components.…”

Section: Introductionmentioning

confidence: 99%

“…Such a fusion approach demonstrates effectiveness for applications like motion capture analysis and topic modeling, where a global posterior is required. In this paper, inspired by work in [2], we extend the MAP inference in PFNM through an additional KL divergence penalty that incorporates local neuron information under the BBP. Doing so allows an inference to exploit the posterior information at both the local and global levels.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information

Xiao¹,

Cheng²

2020

Preprint

View full text Add to dashboard Cite

In federated learning, models trained on local clients are distilled into a global model. Due to the permutation invariance arises in neural networks, it is necessary to match the hidden neurons first when executing federated learning with neural networks. Through the Bayesian nonparametric framework, Probabilistic Federated Neural Matching (PFNM) matches and fuses local neural networks so as to adapt to varying global model size and the heterogeneity of the data. In this paper, we propose a new method which extends the PFNM with a Kullback-Leibler (KL) divergence over neural components product, in order to make inference exploiting posterior information in both local and global levels. We also show theoretically that The additional part can be seamlessly concatenated into the match-and-fuse progress. Through a series of simulations, it indicates that our new method outperforms popular state-of-the-art federated learning methods in both single communication round and additional communication rounds situation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Adaptive Matched Aggregation Formulationmentioning

confidence: 99%

Section: Gpi-fnmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information

Xiao¹,

Cheng²

2020

Preprint

View full text Add to dashboard Cite

show abstract

Bayesian Federated Neural Matching That Completes Full Information

Cheng

2023

AAAI

View full text Add to dashboard Cite

Federated learning is a contemporary machine learning paradigm where locally trained models are distilled into a global model. Due to the intrinsic permutation invariance of neural networks, Probabilistic Federated Neural Matching (PFNM) employs a Bayesian nonparametric framework in the generation process of local neurons, and then creates a linear sum assignment formulation in each alternative optimization iteration. But according to our theoretical analysis, the optimization iteration in PFNM omits global information from existing. In this study, we propose a novel approach that overcomes this flaw by introducing a Kullback-Leibler divergence penalty at each iteration. The effectiveness of our approach is demonstrated by experiments on both image classification and semantic segmentation tasks.

show abstract

Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent

Kassab¹,

Simeone²

2020

Preprint

View full text Add to dashboard Cite

This paper introduces Distributed Stein Variational Gradient Descent (DSVGD), a non-parametric generalized Bayesian inference framework for federated learning. DSVGD maintains a number of non-random and interacting particles at a central server to represent the current iterate of the model global posterior. The particles are iteratively downloaded and updated by one of the agents with the end goal of minimizing the global free energy. By varying the number of particles, DSVGD enables a flexible trade-off between per-iteration communication load and number of communication rounds. DSVGD is shown to compare favorably to benchmark frequentist and Bayesian federated learning strategies, also scheduling a single device per iteration, in terms of accuracy and scalability with respect to the number of agents, while also providing well-calibrated, and hence trustworthy, predictions.

show abstract

Model Fusion with Kullback--Leibler Divergence

Cited by 3 publications

References 0 publications

Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information

Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information

Bayesian Federated Neural Matching That Completes Full Information

Federated Generalized Bayesian Learning via Distributed Stein Variational Gradient Descent

Contact Info

Product

Resources

About