Zero-shot Knowledge Transfer via Adversarial Belief Matching

Micaelli, Paul; Storkey, Amos

doi:10.48550/arxiv.1905.09768

Cited by 12 publications

(22 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One drawback of FedDF and FedBE is that the aggregation server needs to have its own unlabeled dataset. However, this can be overcome using zero-shot learning, where synthetic data can be generated for this purpose Micaelli and Storkey [2019], Nayak et al…”

Section: Model Poising Attacksmentioning

confidence: 99%

FedRAD: Federated Robust Adaptive Distillation

Páll¹,

Samuel²,

Muñoz-González³

et al. 2021

Preprint

View full text Add to dashboard Cite

The robustness of federated learning (FL) is vital for the distributed training of an accurate global model that is shared among large number of clients. The collaborative learning framework by typically aggregating model updates is vulnerable to model poisoning attacks from adversarial clients. Since the shared information between the global server and participants are only limited to model parameters, it is challenging to detect bad model updates. Moreover, real-world datasets are usually heterogeneous and not independent and identically distributed (Non-IID) among participants, which makes the design of such robust FL pipeline more difficult. In this work, we propose a novel robust aggregation method, Federated Robust Adaptive Distillation (FedRAD), to detect adversaries and robustly aggregate local models based on properties of the median statistic, and then performing an adapted version of ensemble Knowledge Distillation. We run extensive experiments to evaluate the proposed method against recently published works. The results show that FedRAD outperforms all other aggregators in the presence of adversaries, as well as in heterogeneous data distributions.

show abstract

Section: Model Poising Attacksmentioning

confidence: 99%

FedRAD: Federated Robust Adaptive Distillation

Páll¹,

Samuel²,

Muñoz-González³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Baselines. Two types of DFKD methods are compared in our experiments: (1) generative methods that train a generative model for synthesis, including DAFL (Chen et al 2019), ZSKT (Micaelli and Storkey 2019), DFQ (Choi et al 2020), and Generative DFD (Luo et al 2020) (2) non-generative methods that craft transfer set in a batch-by-batch manner including DeepInv (Yin et al 2019) and CMI (Fang et al 2021b).…”

Section: Experimental Settingsmentioning

confidence: 99%

Up to 100$\times$ Faster Data-free Knowledge Distillation

Fang¹,

Mo²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Data-free knowledge distillation (DFKD) has recently been attracting increasing attention from research communities, attributed to its capability to compress a model only using synthetic data. Despite the encouraging results achieved, stateof-the-art DFKD methods still suffer from the inefficiency of data synthesis, making the data-free training process extremely time-consuming and thus inapplicable for large-scale tasks. In this work, we introduce an efficacious scheme, termed as FastDFKD, that allows us to accelerate DFKD by a factor of orders of magnitude. At the heart of our approach is a novel strategy to reuse the shared common features in training data so as to synthesize different data instances. Unlike prior methods that optimize a set of data independently, we propose to learn a meta-synthesizer that seeks common features as the initialization for the fast data synthesis. As a result, FastDFKD achieves data synthesis within only a few steps, significantly enhancing the efficiency of data-free training. Experiments over CIFAR, NYUv2, and ImageNet demonstrate that the proposed FastDFKD achieves 10× and even 100× acceleration while preserving performances on par with state of the art.

show abstract

“…Nayak et al [20] proposed a similar scheme but with a zero-shot approach by modeling the output space of the teacher model as a Dirichlet distribution, which is taken from model weights. More recent studies have employed generator architectures similar to GAN [21] to generate synthetic samples replacing the original data [22,23,24,25]. In the absence of the original data for training, DAFL [22] used the teacher model to replace the discriminator by encouraging the outputs to be close to a one-hot distribution and by maximizing the activation counts.…”

Section: Data-free Compressionmentioning

confidence: 99%

“…KegNet [26] adopted a similar idea and used a low-rank decomposition to aid the compression. Adversarial belief matching [23] and data-free adversarial distillation [24] methods suggested adversarially training the generator, such that the generated samples become harder to train. One other variant is to modify samples directly using logit maximization as in DeepInversion [25].…”

Section: Data-free Compressionmentioning

confidence: 99%

Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

Choi¹,

Hong²,

Park³

et al. 2021

Preprint

View full text Add to dashboard Cite

Model quantization is known as a promising method to compress deep neural networks, especially for inferences on lightweight mobile or edge devices. However, model quantization usually requires access to the original training data to maintain the accuracy of the full-precision models, which is often infeasible in real-world scenarios for security and privacy issues. A popular approach to perform quantization without access to the original data is to use synthetically generated samples, based on batch-normalization statistics or adversarial learning. However, the drawback of such approaches is that they primarily rely on random noise input to the generator to attain diversity of the synthetic samples. We find that this is often insufficient to capture the distribution of the original data, especially around the decision boundaries. To this end, we propose Qimera, a method that uses superposed latent embeddings to generate synthetic boundary supporting samples. For the superposed embeddings to better reflect the original distribution, we also propose using an additional disentanglement mapping layer and extracting information from the full-precision model. The experimental results show that Qimera achieves state-of-the-art performances for various settings on data-free quantization. Code is available at https://github.com/iamkanghyunchoi/qimera.

show abstract

Zero-shot Knowledge Transfer via Adversarial Belief Matching

Cited by 12 publications

References 12 publications

FedRAD: Federated Robust Adaptive Distillation

FedRAD: Federated Robust Adaptive Distillation

Up to 100$\times$ Faster Data-free Knowledge Distillation

Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

Contact Info

Product

Resources

About