Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-Tailed Classification

Xiang, Liuyu; Ding, Guiguang; Han, Jungong

doi:10.1007/978-3-030-58558-7_15

Cited by 206 publications

(127 citation statements)

References 48 publications

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…Except for distilling with model compression, knowledge distillation is also proved to be effective when the teacher and the student have identical architectures, i.e., self-distillation [ 37 , 38 ], which transfers the knowledge between the same model structures. Knowledge distillation has also been applied in other areas such as semisupervised learning [ 15 ], curriculum learning [ 39 ], and neural style transfer [ 40 ].…”

Section: Related Workmentioning

confidence: 99%

[Retracted] Relieving the Incompatibility of Network Representation and Classification for Long‐Tailed Data Distribution

Gao

Ming-sheng

2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

In the real-world scenario, data often have a long-tailed distribution and training deep neural networks on such an imbalanced dataset has become a great challenge. The main problem caused by a long-tailed data distribution is that common classes will dominate the training results and achieve a very low accuracy on the rare classes. Recent work focuses on improving the network representation ability to overcome the long-tailed problem, while it always ignores adapting the network classifier to a long-tailed case, which will cause the “incompatibility” problem of network representation and network classifier. In this paper, we use knowledge distillation to solve the long-tailed data distribution problem and fully optimize the network representation and classifier simultaneously. We propose multiexperts knowledge distillation with class-balanced sampling to jointly learn high-quality network representation and classifier. Also, a channel activation-based knowledge distillation method is also proposed to improve the performance further. State-of-the-art performance on several large-scale long-tailed classification datasets shows the superior generalization of our method.

show abstract

Section: Related Workmentioning

confidence: 99%

[Retracted] Relieving the Incompatibility of Network Representation and Classification for Long‐Tailed Data Distribution

Gao

Ming-sheng

2021

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…According to Sweeney and Najafian (2020), the more imbalanced/skewed a prediction produced by a trained model is, the more unfair opportunities it gives over predefined categories, the more unfairly-discriminative the trained model is. We thus follow previous work (Xiang and Ding, 2020;Sweeney and Najafian, 2020) to use the metricimbalance divergence -to evaluate whether a prediction (normally a probability distribution) P is imbalanced/skewed/unfair:…”

Section: Bias Analysismentioning

confidence: 99%

Counterfactual Inference for Text Classification Debiasing

Chen¹,

Feng²,

Wen³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Today's text classifiers inevitably suffer from unintended dataset biases, especially the document-level label bias and word-level keyword bias, which may hurt models' generalization. Many previous studies employed datalevel manipulations or model-level balancing mechanisms to recover unbiased distributions and thus prevent models from capturing the two types of biases. Unfortunately, they either suffer from the extra cost of data collection/selection/annotation or need an elaborate design of balancing strategies. Different from traditional factual inference in which debiasing occurs before or during training, counterfactual inference mitigates the influence brought by unintended confounders after training, which can make unbiased decisions with biased observations. Inspired by this, we propose a model-agnostic text classification debiasing framework -CORSAIR, which can effectively avoid employing data manipulations or designing balancing mechanisms. Concretely, CORSAIR first trains a base model on a training set directly, allowing the dataset biases "poison" the trained model. In inference, given a factual input document, COR-SAIR imagines its two counterfactual counterparts to distill and mitigate the two biases captured by the poisonous model. Extensive experiments demonstrate CORSAIR's effectiveness, generalizability and fairness. 1 * This work was partly done during Chen Qian's internship at Alibaba DAMO academy.

show abstract

“…In contrast to two-phase distillation approaches, on-the-fly Native Ensemble [54] is proposed, which encounters training procedure as a one-stage online distillation. In 2020, Xiang et al [55] proposed a novel framework called Learning From Multiple Experts. In the definitions proposed by authors, 'Experts' refer to the models which extract features on less imbalanced data distribution.…”

Section: B Multi-teacher Kdmentioning

confidence: 99%

Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation

Amirkhani¹,

Khosravian

Masih‐Tehrani

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Recent studies have recently exploited knowledge distillation (KD) technique to address timeconsuming annotation task in semantic segmentation, through which one teacher trained on a single dataset could be leveraged for annotating unlabeled data. However, in this context, knowledge capacity is restricted, and knowledge variety is rare in different conditions, such as cross-model KD, in which the single teacher KD prohibits the student model from distilling information using cross-domain context. To fix this concern, we have looked into learning a lightweight student from a group of teachers. To be more specific, we train five distinct lightweight convolutional neural networks (CNNs) for semantic segmentation on several datasets. Several state-of-the-art augmentation transformations have also been utilized in our training phase. The impacts of such training scenarios are then assessed in terms of student robustness and accuracy. As the main contribution of this paper, our proposed multi-teacher KD paradigm endows the student with the ability to amalgamate and capture a variety of knowledge illustrations from different sources. Results demonstrated that our method outperforms the existing studies on both clean and corrupted data in the semantic segmentation task while benefiting from our proposed score weight system. Experiments validate that our multi-teacher framework results in an improvement of 9% up to 32.18% compared to the singleteacher paradigm. Moreover, it is demonstrated that our paradigm surpasses previous supervised real-time studies in the semantic segmentation challenge.

show abstract

Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-Tailed Classification

Cited by 206 publications

References 48 publications

[Retracted] Relieving the Incompatibility of Network Representation and Classification for Long‐Tailed Data Distribution

[Retracted] Relieving the Incompatibility of Network Representation and Classification for Long‐Tailed Data Distribution

Counterfactual Inference for Text Classification Debiasing

Robust Semantic Segmentation With Multi-Teacher Knowledge Distillation

Contact Info

Product

Resources

About