Distill on the Go: Online knowledge distillation in self-supervised learning

Bhat, Prashant; Zonooz, Bahram

doi:10.1109/cvprw53098.2021.00301

Cited by 30 publications

(20 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We train teacher and student networks with a batch size of 32 and 100 epochs. Our teacher model achieves a similar accuracy with literature [5] (52.02% vs. 48.26%). We train CVAE in the same way as in CIFAR-100.…”

Section: Results On Tiny-imagenetsupporting

confidence: 81%

Black-box Few-shot Knowledge Distillation

Nguyen¹,

Gupta²,

Do³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge distillation (KD) is an efficient approach to transfer the knowledge from a large "teacher" network to a smaller "student" network. Traditional KD methods require lots of labeled training samples and a white-box teacher (parameters are accessible) to train a good student. However, these resources are not always available in real-world applications. The distillation process often happens at an external party side where we do not have access to much data, and the teacher does not disclose its parameters due to security and privacy concerns. To overcome these challenges, we propose a black-box few-shot KD method to train the student with few unlabeled training samples and a black-box teacher. Our main idea is to expand the training set by generating a diverse set of out-of-distribution synthetic images using MixUp and a conditional variational auto-encoder. These synthetic images along with their labels obtained from the teacher are used to train the student. We conduct extensive experiments to show that our method significantly outperforms recent SOTA few/zero-shot KD methods on image classification tasks. The code and models are available at: https://github.com/nphdang/FS-BBT

show abstract

Section: Results On Tiny-imagenetsupporting

confidence: 81%

Black-box Few-shot Knowledge Distillation

Nguyen¹,

Gupta²,

Do³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, [19] first uses self-distillation, which transfers the knowledge in the deeper portion of the networks to the shallow sections. [20] improves representation quality of the smaller models by single-stage online knowledge distillation. [21] uses self-distillation between different epochs with soft targets.…”

Section: Self-distillationmentioning

confidence: 99%

Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition

Zhang¹,

He²,

Lü³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video-based action recognition is one of the most popular topics in computer vision. With recent advances of selfsupervised video representation learning approaches, action recognition usually follows a two-stage training framework, i.e., self-supervised pre-training on large-scale unlabeled sets and transfer learning on a downstream labeled set. However, catastrophic forgetting of the pre-trained knowledge becomes the main issue in the downstream transfer learning of action recognition, resulting in a sub-optimal solution. In this paper, to alleviate the above issue, we propose a novel transfer learning approach that combines self-distillation in fine-tuning to preserve knowledge from the pre-trained model learned from the large-scale dataset. Specifically, we fix the encoder from the last epoch as the teacher model to guide the training of the encoder from the current epoch in the transfer learning. With such a simple yet effective learning strategy, we outperform state-of-the-art methods on widely used UCF101 and HMDB51 datasets in action recognition task.

show abstract

“…To ensure fairness of our comparisons, we apply the new training protocol to both the previous baseline AL models, and to our proposed DAES framework. The proposed training protocol could help to eventually unify this line of work with some form of knowledge distillation [46,47].…”

Section: Daes With Self-trained Knowledge Distillationmentioning

confidence: 99%

Deep Active Ensemble Sampling For Image Classification

Doretto¹,

Adjeroh²

2022

Preprint

View full text Add to dashboard Cite

Conventional active learning (AL) frameworks aim to reduce the cost of data annotation by actively requesting the labeling for the most informative data points. However, introducing AL to data hungry deep learning algorithms has been a challenge. Some proposed approaches include uncertainty-based techniques, geometric methods, implicit combination of uncertainty-based and geometric approaches, and more recently, frameworks based on semi/self supervised techniques. In this paper, we address two specific problems in this area. The first is the need for efficient exploitation/exploration trade-off in sample selection in AL. For this, we present an innovative integration of recent progress in both uncertainty-based and geometric frameworks to enable an efficient exploration/exploitation trade-off in sample selection strategy. To this end, we build on a computationally efficient approximate of Thompson sampling with key changes as a posterior estimator for uncertainty representation. Our framework provides two advantages: (1) accurate posterior estimation, and ( 2) tune-able trade-off between computational overhead and higher accuracy. The second problem is the need for improved training protocols in deep AL. For this, we use ideas from semi/self supervised learning to propose a general approach that is independent of the specific AL technique being used. Taken these together, our framework shows a significant improvement over the state-of-the-art, with results that are comparable to the performance of supervised-learning under the same setting. We show empirical results of our framework, and comparative performance with the state-of-the-art on four datasets, namely, MNIST, CIFAR10, CIFAR100 and ImageNet to establish a new baseline in two different settings.

show abstract

Distill on the Go: Online knowledge distillation in self-supervised learning

Cited by 30 publications

References 4 publications

Black-box Few-shot Knowledge Distillation

Black-box Few-shot Knowledge Distillation

Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition

Deep Active Ensemble Sampling For Image Classification

Contact Info

Product

Resources

About