Similarity-Preserving Knowledge Distillation

Tung, Fred; Mori, Greg

doi:10.1109/iccv.2019.00145

Cited by 818 publications

(496 citation statements)

References 33 publications

Supporting

Mentioning

492

Contrasting

Unclassified

Order By: Relevance

“…There is active research on methods that can produce more compact networks with improved prediction capability. Common approaches include knowledge distillation [206], where a compact student network is trained to mimic a larger network, e.g., by guiding the network to produce similar activations for similar inputs, and advanced network models, such as Operational Neural Networks [207], where the linear operators of CNNs are replaced by various (non-)linear operations, which allows to produce complex outputs which much fewer parameters. [208], [209].…”

Section: Fast and Computationally Light Methodsmentioning

confidence: 99%

Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision

et al. 2020

View full text Add to dashboard Cite

Search and rescue (SAR) operations can take significant advantage from supporting autonomous or teleoperated robots and multi-robot systems. These can aid in mapping and situational assessment, monitoring and surveillance, establishing communication networks, or searching for victims. This paper provides a review of multi-robot systems supporting SAR operations, with system-level considerations and focusing on the algorithmic perspectives for multi-robot coordination and perception. This is, to the best of our knowledge, the first survey paper to cover (i) heterogeneous SAR robots in different environments, (ii) active perception in multi-robot systems, while (iii) giving two complementary points of view from the multi-agent perception and control perspectives. We also discuss the most significant open research questions: shared autonomy, sim-to-real transferability of existing methods, awareness of victims' conditions, coordination and interoperability in heterogeneous multi-robot systems, and active perception. The different topics in the survey are put in the context of the different challenges and constraints that various types of robots (ground, aerial, surface, or underwater) encounter in different SAR environments (maritime, urban, wilderness, or other post-disaster scenarios). The objective of this survey is to serve as an entry point to the various aspects of multi-robot SAR systems to researchers in both the machine learning and control fields by giving a global overview of the main approaches being taken in the SAR robotics area.

show abstract

Section: Fast and Computationally Light Methodsmentioning

confidence: 99%

Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision

et al. 2020

View full text Add to dashboard Cite

show abstract

“…We selected 8 SOTA methods as baselines for the conventional full-category knowledge distillation task, and compare 8 of them for the partial-category classification task. The involved methods include: Knowledge Distillation (KD) [2], FitNet [9], Attention Transfer (AT) [10], Similarity-Preserving Knowledge Distillation (SP) [34], Correlation Congruence (CC) [15], Variational information distillation (VID) [35], Relational Knowledge Distillation (RKD) [11], and Contrastive Representational Distillation (CRD) [16]. Table I shows the Top-1 accuracy of the fog networks obtained as student using all the baselines mentioned above, classifying a randomly sampled subset of CIFAR100 classes (average on 20,50,70 classes).…”

Section: B Comparison 1: Hbs Cloud-to-fog Vs Modern Supervised Knowmentioning

confidence: 99%

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Wang

Yang

Papanastasiou

et al. 2021

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Deep convoloutional networks have been widely deployed in modern cyber-physical systems performing different visual classification tasks. As the fog and edge devices have different computing capacity and perform different sub-tasks, models trained for one device may not be deployable on another. Knowledge distillation technique can effectively compress well trained convolutional neural networks (CNN) into lightweight models suitable to different devices. However, due to privacy issue and transmission cost, manually annotated data for training the deep learning models are usually gradually collected and archived in different sites. Simply training a model on powerful cloud servers and compressing them for particular edge devices failed to use the distributed data stored at different sites. This offline training approach is also inefficient to deal with new data collected from the edge devices. To overcome these obstacles, we propose the heterogeneous brain storming (HBS) method for object recognition tasks in real-world IoT scenarios. Our method enables flexible bidirectional federated learning of heterogeneous models trained on distributed datasets with a new "brain storming" mechanism and optimizable temperature parameters. In our comparison experiments, this heterogeneous brain storming method outperformed multiple state-of-the-art single-model compression methods, as well as the newest multinetwork knowledge distillation methods with both homogeneous and heterogeneous classifiers. The ablation experiment results proved that the trainable temperature parameter into the conventional knowledge distillation loss can effectively ease the learning process of student networks in different methods. To the best of

show abstract

“…Relation-based knowledge does not focus on the value of certain layers but explores the relationship between different sample data or network feature layers. In our paper, similarity-preserving knowledge distillation method [30] is used, as is shown in Fig. 6.…”

Section: Response-based Knowledgementioning

confidence: 99%

Knowledge from the original network: restore a better pruned network with knowledge distillation

Chen

et al. 2021

Complex Intell. Syst.

View full text Add to dashboard Cite

To deploy deep neural networks to edge devices with limited computation and storage costs, model compression is necessary for the application of deep learning. Pruning, as a traditional way of model compression, seeks to reduce the parameters of model weights. However, when a deep neural network is pruned, the accuracy of the network will significantly decrease. The traditional way to decrease the accuracy loss is fine-tuning. When over many parameters are pruned, the pruned network’s capacity is reduced heavily and cannot recover to high accuracy. In this paper, we apply the knowledge distillation strategy to abate the accuracy loss of pruned models. The original network of the pruned network was used as the teacher network, aiming to transfer the dark knowledge from the original network to the pruned sub-network. We have applied three mainstream knowledge distillation methods: response-based knowledge, feature-based knowledge, and relation-based knowledge (Gou et al. in Knowledge distillation: a survey. arXiv:200605525, 2020), and compare the result to the traditional fine-tuning method with grand-truth labels. Experiments have been done on the CIFAR100 dataset with several deep convolution neural network. Results show that the pruned network recovered by knowledge distillation with its original network performs better accuracy than it recovered by fine-tuning with sample labels. It has also been validated in this paper that the original network as the teacher performs better than differently structured networks with same accuracy as the teacher.

show abstract

Similarity-Preserving Knowledge Distillation

Cited by 818 publications

References 33 publications

Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision

Collaborative Multi-Robot Search and Rescue: Planning, Coordination, Perception, and Active Vision

Industrial Cyber-Physical Systems-Based Cloud IoT Edge for Federated Heterogeneous Distillation

Knowledge from the original network: restore a better pruned network with knowledge distillation

Contact Info

Product

Resources

About