Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning

Abstract

We propose a new method for unsupervised generative continual learning through realignment of Variational Autoencoder's latent space. Deep generative models suffer from catastrophic forgetting in the same way as other neural structures. Recent generative continual learning works approach this problem and try to learn from new data without forgetting previous knowledge. However, those methods usually focus on artificial scenarios where examples share almost no similarity between subsequent portions of data - an assumption not realistic in the real-life applications of continual learning. In this work, we identify this limitation and posit the goal of generative continual learning as a knowledge accumulation task. We solve it by continuously aligning latent representations of new data that we call bands in additional latent space where examples are encoded independently of their source task. In addition, we introduce a method for controlled forgetting of past data that simplifies this process. On top of the standard continual learning benchmarks, we propose a novel challenging knowledge consolidation scenario and show that the proposed approach outperforms state-of-the-art by up to twofold across all experiments and additional real-life evaluation. To our knowledge, Multiband VAE is the first method to show forward and backward knowledge transfer in generative continual learning.

Survey of Knowledge Distillation in Federated Edge Learning

“…Knowledge Type Require Public Dataset STU-KD [44] Logits Yes DS-FL [13] FD [14] CFD [31] FedGEMS [6] Fed-ET [7] FedKEMF [41] [30] Mix2FLD [28] Ahn, et al [1,2] Def-KT [19] CMFD [34] pFedSD [17] No FedZKT [43] NRFL [25] FedGKT [10] Features+Logits No FedICT [39] FedD3 [32] Distilled Datasets No cation efficiency improvement compared with prior works. On top of this, Song [32] distils local datasets on devices and uploads the distilled dataset to the edge server, requiring only one-shot communication during the entire training process.…”

“…To reduce on-device computation overhead in FEL, a series of approaches [10,6,7] are proposed to enable devices training much smaller models than the edge server, leveraging KD as an exchange protocol across model representations. Specifically, He [10] and Cheng [6] establish alternating minimization FEL frameworks to transfer knowledge from compact on-device models to the large edge model via KD, after which the on-device models are optimized based on the knowledge transferred back from the edge.…”

“…Specifically, He [10] and Cheng [6] establish alternating minimization FEL frameworks to transfer knowledge from compact on-device models to the large edge model via KD, after which the on-device models are optimized based on the knowledge transferred back from the edge. In addition, Cho [7] proposes an ensemble knowledge transfer algorithm for collaborative training across devices, where the large edge model learns the weighted consensus of the uploaded small on-device models via KD.…”

“…In addition, Qi [30] employs KD after on-device model uploading and edge model downloading, and uses a conditional variational autoencoder to complement both the required public data and private data. It is also worth noting that previously mentioned approaches taking KD as the end-edge model representation exchange protocol [10,6,7,39,13,14] also support heterogeneous on-device models, which are partly applicable on heterogeneous devices as well.…”

“…Recent trends suggest the great potential to address the above challenges in the context of FEL. Previous works integrating KD into the training process of FEL have been successful in tackling constrained device resources [13,14,31,32,6,10,7], adapting to heterogeneous devices and user requirements [39,17,44,43,41,30,25], and adapting to complex communication channels as well as network topologies [1,2,28,19,34]. Therefore, a survey is urgently needed to review how KD applies to FEL.…”