Correlation Congruence for Knowledge Distillation

Peng, Baoyun; Xiao, Jin; Li, Dongsheng; Zhou, Shunfeng; Wu, Yichao; Liu, Jiaheng; Zhang, Zhaoning; Liu, Yu

doi:10.1109/iccv.2019.00511

Cited by 471 publications

(249 citation statements)

References 31 publications

Supporting

Mentioning

246

Contrasting

Unclassified

Order By: Relevance

“…We selected 8 SOTA methods as baselines for the conventional full-category knowledge distillation task, and compare 8 of them for the partial-category classification task. The involved methods include: Knowledge Distillation (KD) [2], FitNet [9], Attention Transfer (AT) [10], Similarity-Preserving Knowledge Distillation (SP) [34], Correlation Congruence (CC) [15], Variational information distillation (VID) [35], Relational Knowledge Distillation (RKD) [11], and Contrastive Representational Distillation (CRD) [16]. Table I shows the Top-1 accuracy of the fog networks obtained as student using all the baselines mentioned above, classifying a randomly sampled subset of CIFAR100 classes (average on 20,50,70 classes).…”

Section: B Comparison 1: Hbs Cloud-to-fog Vs Modern Supervised Knowmentioning

confidence: 99%

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Wang

Yang

Papanastasiou

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different sub-tasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks (CNN) into lightweight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world IoT scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new "brain storming" mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of

show abstract

Section: B Comparison 1: Hbs Cloud-to-fog Vs Modern Supervised Knowmentioning

confidence: 99%

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Wang

Yang

Papanastasiou

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

show abstract

“…Many methods have been proposed to minimize the performance gap between a student and a teacher. We discuss different forms of knowledge in the following categories: response-based knowledge [26,27,35], feature-based knowledge [28,29,[36][37][38][39][40][41][42][43][44][45], and relation-based knowledge [30,31,[46][47][48][49].…”

Section: Knowledge Distillationmentioning

confidence: 99%

“…The work in [48] proposed a Similarity Preserving (SP) distillation method to transfer pairwise activation similarities of input samples. Peng et al proposed a method based on CC [31], in which the distilled knowledge contains information of instances and the correlations between two instances. Liu et al proposed Instance Relationship Graph (IRG) [49] which contains instance features and relationships, and the feature space transformation cross layers as the knowledge to transfer.…”

Section: Relation-based Knowledgementioning

confidence: 99%

“…For instance, Park et al proposed Relational Knowledge Distillation (RKD) [30] to transfer relations of data examples based on distance-wise and angle-wise distillation losses which measure structural differences in relations. Peng et al [31] proposed a method based on correlation congruence (CC), in which the distilled knowledge contains information of instances and the correlations between two instances.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ensemble Learning of Lightweight Deep Learning Models Using Knowledge Distillation for Image Classification

Kang

Gwak

2020

Mathematics

View full text Add to dashboard Cite

In recent years, deep learning models have been used successfully in almost every field including both industry and academia, especially for computer vision tasks. However, these models are huge in size, with millions (and billions) of parameters, and thus cannot be deployed on the systems and devices with limited resources (e.g., embedded systems and mobile phones). To tackle this, several techniques on model compression and acceleration have been proposed. As a representative type of them, knowledge distillation suggests a way to effectively learn a small student model from large teacher model(s). It has attracted increasing attention since it showed its promising performance. In the work, we propose an ensemble model that combines feature-based, response-based, and relation-based lightweight knowledge distillation models for simple image classification tasks. In our knowledge distillation framework, we use ResNet−20 as a student network and ResNet−110 as a teacher network. Experimental results demonstrate that our proposed ensemble model outperforms other knowledge distillation models as well as the large teacher model for image classification tasks, with less computational power than the teacher model.

show abstract

“…By combining the ideas of the generative adversarial network (GAN) [47] and knowledge distillation, the student network is able to effectively learn the performance of the teacher network [48]. In [49], the authors proposed a knowledge distillation method based on the correlation between instances. Unlike the previous knowledge distillation method, the correlation congruence knowledge distillation (CCKD) method transfers not only the instance-level information, but also the correlation between instances.…”

Section: Knowledge Distillationmentioning

confidence: 99%

Compressing Deep Networks by Neuron Agglomerative Clustering

Wang

Liu

et al. 2020

Sensors

View full text Add to dashboard Cite

In recent years, deep learning models have achieved remarkable successes in various applications, such as pattern recognition, computer vision, and signal processing. However, high-performance deep architectures are often accompanied by a large storage space and long computational time, which make it difficult to fully exploit many deep neural networks (DNNs), especially in scenarios in which computing resources are limited. In this paper, to tackle this problem, we introduce a method for compressing the structure and parameters of DNNs based on neuron agglomerative clustering (NAC). Specifically, we utilize the agglomerative clustering algorithm to find similar neurons, while these similar neurons and the connections linked to them are then agglomerated together. Using NAC, the number of parameters and the storage space of DNNs are greatly reduced, without the support of an extra library or hardware. Extensive experiments demonstrate that NAC is very effective for the neuron agglomeration of both the fully connected and convolutional layers, which are common building blocks of DNNs, delivering similar or even higher network accuracy. Specifically, on the benchmark CIFAR-10 and CIFAR-100 datasets, using NAC to compress the parameters of the original VGGNet by 92.96% and 81.10%, respectively, the compact network obtained still outperforms the original networks.

show abstract

Correlation Congruence for Knowledge Distillation

Cited by 471 publications

References 31 publications

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Ensemble Learning of Lightweight Deep Learning Models Using Knowledge Distillation for Image Classification

Compressing Deep Networks by Neuron Agglomerative Clustering

Contact Info

Product

Resources

About