Generalization bounds via distillation

Hsu, Daniel; Ji, Ziwei; Telgarsky, Matus; Wang, Lan

doi:10.48550/arxiv.2104.05641

Cited by 2 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different to the classical bound, we compress the hypothetical space F into a smaller space derived by the teacher feature extractor. Hsu et al [37] points out that the distillation helps to derive a finegrained analysis for the prediction risk and avoid getting a vacuous generalization bound. Therefore, we take the following theorem as a starting point to derive robustness transfer.…”

Section: Theoretical Analysismentioning

confidence: 99%

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

Awais¹,

Zhou²,

Xie³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks are susceptible to adversarially crafted, small and imperceptible changes in the natural inputs. The most effective defense mechanism against these examples is adversarial training which constructs adversarial examples during training by iterative maximization of loss. The model is then trained to minimize the loss on these constructed examples. This min-max optimization requires more data, larger capacity models, and additional computing resources. It also degrades the standard generalization performance of a model. Can we achieve robustness more efficiently? In this work, we explore this question from the perspective of knowledge transfer. First, we theoretically show the transferability of robustness from an adversarially trained teacher model to a student model with the help of mixup augmentation. Second, we propose a novel robustness transfer method called Mixup-Based Activated Channel Maps (MixACM) Transfer. MixACM transfers robustness from a robust teacher to a student by matching activated channel maps generated without expensive adversarial perturbations. Finally, extensive experiments on multiple datasets and different learning scenarios show our method can transfer robustness while also improving generalization on natural images.

show abstract

Section: Theoretical Analysismentioning

confidence: 99%

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

Awais¹,

Zhou²,

Xie³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

On Predicting Generalization using GANs

Zhang¹,

Gupta²,

Saunshi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Research on generalization bounds for deep networks seeks to give ways to predict test error using just the training dataset and the network parameters. While generalization bounds can give many insights about architecture design, training algorithms etc., what they do not currently do is yield good predictions for actual test error. A recently introduced Predicting Generalization in Deep Learning competition (Jiang et al., 2020) aims to encourage discovery of methods to better predict test error. The current paper investigates a simple idea: can test error be predicted using synthetic data, produced using a Generative Adversarial Network (GAN) that was trained on the same training dataset? Upon investigating several GAN models and architectures, we find that this turns out to be the case. In fact, using GANs pre-trained on standard datasets, the test error can be predicted without requiring any additional hyperparameter tuning. This result is surprising because GANs have well-known limitations (e.g. mode collapse) and are known to not learn the data distribution accurately. Yet the generated samples are good enough to substitute for test data. Several additional experiments are presented to explore reasons why GANs do well at this task. In addition to a new approach for predicting generalization, the counter-intuitive phenomena presented in our work may also call for a better understanding of GANs' strengths and limitations.

show abstract

Generalization bounds via distillation

Cited by 2 publications

References 16 publications

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Maps

On Predicting Generalization using GANs

Contact Info

Product

Resources

About