Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

Ridnik, Tal; Lawen, Hussam; Ben-Baruch, Emanuel; Noy, Asaf

doi:10.48550/arxiv.2204.03475

Cited by 4 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, we can mix both the hard and soft target distillation by setting α to be between 0 and 1. We can also mix soft target distillation with supervised training by using LRNNT on labeled data and LKL on unlabeled data, which is used in existing distillation work in other domains [4,26,27]. In this paper, most of the experiments use α = 0 because we found that using LKL alone achieves better WERs, as shown in Section 4.3.…”

Section: Distillation Methodsmentioning

confidence: 95%

Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR

Hwang¹,

Sim²,

Strohman³

2022

Preprint

View full text Add to dashboard Cite

Knowledge distillation is an effective machine learning technique to transfer knowledge from a teacher model to a smaller student model, especially with unlabeled data. In this paper, we focus on knowledge distillation for the RNN-T model, which is widely used in state-ofthe-art (SoTA) automatic speech recognition (ASR). Specifically, we compared using soft and hard target distillation to train large-scale RNN-T models on the LibriSpeech/LibriLight public dataset (60k hours) and our in-house data (600k hours). We found that hard targets are more effective when the teacher and student have different architecture, such as large teacher and small streaming student. On the other hand, soft target distillation works better in self-training scenario like iterative large teacher training. For a large model with 0.6B weights, we achieve a new SoTA word error rate (WER) on LibriSpeech (8% relative improvement on dev-other) using Noisy Student Training with soft target distillation. It also allows our production teacher to adapt new data domain continuously.

show abstract

Section: Distillation Methodsmentioning

confidence: 95%

Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR

Hwang¹,

Sim²,

Strohman³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Different from the result that using only teacher's prediction (i.e. α kd = ∞) can achieve top results [33], ground-truth is still needed in GenSCL. A teacher classifier with better performances could potentially improve results better.…”

Section: Kd Teacher Relative Weightmentioning

confidence: 94%

“…In addition to cross-entropy, Kullback-Leibler divergence between student's prediction and teacher's prediction can boost the training of the student. A recent study [33] has shown that the training scheme based on modern tricks and knowledge distillation can stably achieve top results on cross-entropy. In contrastive learning paradigm, contrastive representation distillation [39] has shown that using teacher's representions to guide teacher model Figure 2: Generalized supervised contrastive learning framework, GenSCL.…”

Section: Related Workmentioning

confidence: 99%

“…Inspired by USI [33], GenSCL successfully adapts image-based regularizations (Mixup [46] and Cutmix [45], and knowledge-distillation (KD [21]) to supervised contrastive learning (SupCon [23]). As illustrated in Figure 2, our framework comprises the following main components:…”

Section: Genscl: Generalized Contrastive Learning Frameworkmentioning

confidence: 99%

“…Regularization tricks to improve generalization in deep neural nets have been continuously developed. Recently, [33] proposed the Unified Scheme for ImageNet (USI) to train top results on a wide variety Figure 1: Overview of supervised contrasitve loss vs. generalized supervised contrastive loss with knowledge distillation and Cutmix. The supervised contrastive loss (upper, Eq.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A Generalized Supervised Contrastive Learning Framework

Kim¹,

Lee²,

Chang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Based on recent remarkable achievements of contrastive learning in self-supervised representation learning, supervised contrastive learning (SupCon) has successfully extended the batch contrastive approaches to the supervised context and outperformed cross-entropy on various datasets on ResNet. In this work, we present GenSCL: a generalized supervised contrastive learning framework that seamlessly adapts modern image-based regularizations (such as Mixup-Cutmix) and knowledge distillation (KD) to SupCon by our generalized supervised contrastive loss. Generalized supervised contrastive loss is a further extension of supervised contrastive loss measuring cross-entropy between the similarity of labels and that of latent features. Then a model can learn to what extent contrastives should be pulled closer to an anchor in the latent space. By explicitly and fully leveraging label information, GenSCL breaks the boundary between conventional positives and negatives, and any kind of pre-trained teacher classifier can be utilized. ResNet-50 trained in GenSCL with Mixup-Cutmix and KD achieves state-of-the-art accuracies of 97.6% and 84.7% on CIFAR10 and CIFAR100 without external data, which significantly improves the results reported in the original SupCon (1.6% and 8.2%, respectively). Pytorch implementation is available at https://t.ly/yuUO.Preprint. Under review.

show abstract

RCKD: Response-Based Cross-Task Knowledge Distillation for Pathological Image Analysis

Kim,

Kwak,

Chang

et al. 2023

Bioengineering

View full text Add to dashboard Cite

We propose a novel transfer learning framework for pathological image analysis, the Response-based Cross-task Knowledge Distillation (RCKD), which improves the performance of the model by pretraining it on a large unlabeled dataset guided by a high-performance teacher model. RCKD first pretrains a student model to predict the nuclei segmentation results of the teacher model for unlabeled pathological images, and then fine-tunes the pretrained model for the downstream tasks, such as organ cancer sub-type classification and cancer region segmentation, using relatively small target datasets. Unlike conventional knowledge distillation, RCKD does not require that the target tasks of the teacher and student models be the same. Moreover, unlike conventional transfer learning, RCKD can transfer knowledge between models with different architectures. In addition, we propose a lightweight architecture, the Convolutional neural network with Spatial Attention by Transformers (CSAT), for processing high-resolution pathological images with limited memory and computation. CSAT exhibited a top-1 accuracy of 78.6% on ImageNet with only 3M parameters and 1.08 G multiply-accumulate (MAC) operations. When pretrained by RCKD, CSAT exhibited average classification and segmentation accuracies of 94.2% and 0.673 mIoU on six pathological image datasets, which is 4% and 0.043 mIoU higher than EfficientNet-B0, and 7.4% and 0.006 mIoU higher than ConvNextV2-Atto pretrained on ImageNet, respectively.

show abstract

Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

Cited by 4 publications

References 0 publications

Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR

Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR

A Generalized Supervised Contrastive Learning Framework

RCKD: Response-Based Cross-Task Knowledge Distillation for Pathological Image Analysis

Contact Info

Product

Resources

About