Distilling and Transferring Knowledge via cGAN-generated Samples for Image Classification and Regression

Abstract

Knowledge distillation (KD) has been actively studied for image classification tasks in deep learning, aiming to improve the performance of a lightweight (student) model based on the knowledge from a heavyweight (teacher) model. However, applying KD in image regression with a scalar response variable has been rarely studied, and there exists no KD method applicable to both classification and regression tasks yet. Moreover, existing KD methods often require a practitioner to carefully select or adjust the teacher and student architectures, making these methods less scalable in practice. Furthermore, although KD is usually conducted in scenarios with limited labeled data, very few techniques are developed to alleviate such data insufficiency. To address the above concerns in a unified way, we propose a comprehensive KD framework based on conditional generative adversarial networks (cGANs), termed cGAN-KD. Fundamentally different from existing KD methods, cGAN-KD distills and transfers knowledge from a teacher model to a student model via cGAN-generated samples. This novel mechanism makes cGAN-KD suitable for both classification and regression tasks, compatible with other KD methods, and insensitive to the teacher and student architectures. Also, benefiting from recent advances in cGAN and our specially designed subsampling and filtering procedures, the proposed cGAN-KD also performs well when labeled data are limited. An error bound for a student model trained in the cGAN-KD framework is derived in this work, providing a theory for why cGAN-KD is effective as well as guiding the practical implementation of cGAN-KD. Extensive experiments on CIFAR-10 and Tiny-ImageNet show that we can incorporate state-of-the-art KD methods into the cGAN-KD framework to yield a new state of the art. Also, experiments on RC-49 and UTKFace demonstrate the effectiveness of cGAN-KD in image regression tasks, where existing KD methods are inapplicable. Notably, cGAN-KD still works well even if only 20% CIFAR-10 and 10% RC-49 data are available for training.

Learning Proposals for Practical Energy-Based Regression

“…In contrast, our joint training method is non-adversarial and can even be implemented by directly minimizing one unified objective. On a conceptual level, our MDN training approach is also related to work on teacher-student networks and knowledge distillation [27,40,61,8]. In a knowledge distillation problem, a teacher network is utilized to improve the performance of a more lightweight student network.…”

“…In a knowledge distillation problem, a teacher network is utilized to improve the performance of a more lightweight student network. While knowledge distillation for regression is not a particularly well-studied topic, it has been studied for image-based regression tasks in very recent work [8]. A student network is there enhanced by augmenting its training set with images and pseudo targets generated by a conditional GAN and a pre-trained teacher network, respectively.…”

“…during training via the loss J MDN (φ) in(8) should help mitigate some of the known inefficiencies of MDN training. We note that our proposed joint training approach is twice as slow compared to conventional MDN training, as two separate DNNs f θ and g φ are updated at each iteration.…”

SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images

“…Xu et al (Xu, Liu, Li and Loy, 2020) further found that self-supervision signals can effectively transfer the hidden information from the teacher to the student via their proposed SSKD architecture, which substantially benefited the scenarios with few-shot and noisy-labels. Ding et al (Ding, Wang, Xu, Wang and Welch, 2021c) proposed an data augmentationbased knowledge distillation framework for classification and regression tasks via synthetic samples (Ding, Wang, Xu, Welch and Wang, 2021d;Ding, Wang, Wang and Welch, 2021b).…”