DeGAN: Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier

Addepalli, Sravanti; Nayak, Gaurav Kumar; Chakraborty, Anirban; Radhakrishnan, Venkatesh Babu

doi:10.1609/aaai.v34i04.5709

Cited by 35 publications

(30 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we discuss how the inclusion of each technique we propose and hyper-parameter choices related to them, affect distillation performance. We consider DAFL [4], DFAD [6], EATSKD [20], DeGAN [1] and RD-SKD [10] as our baselines for comparison using the same student and teacher models. It is noted, to test the baselines on the benchmarks that they had not reported results on, we used the code on their GitHub repositories.…”

Section: Methodsmentioning

confidence: 99%

Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data

Kuluhan¹,

Pham²,

Mitra³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the increasing popularity of deep learning on edge devices, compressing large neural networks to meet the hardware requirements of resource-constrained devices became a significant research direction. Numerous compression methodologies are currently being used to reduce the memory sizes and energy consumption of neural networks. Knowledge distillation (KD) is among such methodologies and it functions by using data samples to transfer the knowledge captured by a large model (teacher) to a smaller one (student). However, due to various reasons, the original training data might not be accessible at the compression stage. Therefore, data-free model compression is an ongoing research problem that has been addressed by various works. In this paper, we point out that catastrophic forgetting is a problem that can potentially be observed in existing data-free distillation methods. Moreover, the sample generation strategies in some of these methods could result in a mismatch between the synthetic and real data distributions. To prevent such problems, we propose a data-free KD framework that maintains a dynamic collection of generated samples over time. Additionally, we add the constraint of matching the real data distribution in sample generation strategies that target maximum information gain. Our experiments demonstrate that we can improve the accuracy of the student models obtained via KD when compared with state-of-the-art approaches on the SVHN, Fashion MNIST and CIFAR100 datasets.

show abstract

Section: Methodsmentioning

confidence: 99%

Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data

Kuluhan¹,

Pham²,

Mitra³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…by generating underrepresented grades in a risk assessment scoring system [89] for prostate cancer. A further promising applicable method is to enrich the data using a related domain as proxy input [90]. Towards the goal of a more diverse distribution of data with respect to gender and race, similar principles can be applied.…”

Section: Imbalanced Data and Fairnessmentioning

confidence: 99%

A Review of Generative Adversarial Networks in Cancer Imaging: New Applications, New Solutions

Osuala¹,

Kushibar²,

Garrucho³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite technological and medical advances, the detection, interpretation, and treatment of cancer based on imaging data continue to pose significant challenges. These include high inter-observer variability, difficulty of small-sized lesion detection, nodule interpretation and malignancy determination, inter-and intra-tumour heterogeneity, class imbalance, segmentation inaccuracies, and treatment effect uncertainty. The recent advancements in Generative Adversarial Networks (GANs) in computer vision as well as in medical imaging may provide a basis for enhanced capabilities in cancer detection and analysis. In this review, we assess the potential of GANs to address a number of key challenges of cancer imaging, including data scarcity and imbalance, domain and dataset shifts, data access and privacy, data annotation and quantification, as well as cancer detection, tumour profiling and treatment planning. We provide a critical appraisal of the existing literature of GANs applied to cancer imagery, together with suggestions on future research directions to address these challenges. We analyse and discuss 163 papers that apply adversarial training techniques in the context of cancer imaging and elaborate their methodologies, advantages and limitations. With this work, we strive to bridge the gap between the needs of the clinical cancer imaging community and the current and prospective research on GANs in the artificial intelligence community.

show abstract

“…[4,46] Some train a GAN from the teachers to maximize chosen class predictions [47,48,31], sometimes also batchnorm statistics [5,49], and sometimes on proxy data instead. [50,51] These works do not concern either the transferability of the learned student network or merging multiple teachers.…”

Section: Related Workmentioning

confidence: 99%

Representation Consolidation for Training Expert Students

Li,

Ravichandran,

Fowlkes

et al. 2021

Preprint

View full text Add to dashboard Cite

Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher. A more useful goal than emulation, yet under-explored, is for the student to learn feature representations that transfer well to future tasks. However, we observe that standard distillation of task-specific teachers actually reduces the transferability of student representations to downstream tasks. We show that a multi-head, multi-task distillation method using an unlabeled proxy dataset and a generalist teacher is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance, outperforming the teacher(s) and the strong baseline of ImageNet pretrained features. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model, whose representation is improved on all teachers' domain(s).Preprint. Under review.

show abstract

DeGAN: Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier

Cited by 35 publications

References 9 publications

Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data

Preventing Catastrophic Forgetting and Distribution Mismatch in Knowledge Distillation via Synthetic Data

A Review of Generative Adversarial Networks in Cancer Imaging: New Applications, New Solutions

Representation Consolidation for Training Expert Students

Contact Info

Product

Resources

About