Online Knowledge Distillation via Collaborative Learning

Guo, Qiushan; Wang, Xinjiang; Wu, Yichao; Yu, Zhipeng; Ding, Liang; Hu, Xiaolin; Luo, Ping

doi:10.1109/cvpr42600.2020.01103

Cited by 247 publications

(133 citation statements)

References 14 publications

Supporting

Mentioning

132

Contrasting

Unclassified

Order By: Relevance

“…Another ensemble knowledge distillation method was proposed by Guo et al (2020) named Knowledge Distillation via Collaborative Learning (KDCL). KDCL trains on input data that is distorted differently for each student in the ensemble.…”

Section: Surveymentioning

confidence: 99%

Knowledge distillation in deep learning and its applications

Alkhulaifi¹,

Alsahli²,

Ahmad³

2021

PeerJ Computer Science

View full text Add to dashboard Cite

Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present an outlook of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric which compares different knowledge distillation solutions based on models' sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper including the current challenges and possible research directions.

show abstract

Section: Surveymentioning

confidence: 99%

Knowledge distillation in deep learning and its applications

Alkhulaifi¹,

Alsahli²,

Ahmad³

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…The current digital watermarking can also be validated by applying it to protect the copyrights of trained neural networks where ownership protection and piracy prevention is of utmost priority. The traditional approaches in digital watermarking are not fit for the digital data that can be stored efficiently and with very high quality and manipulated easily using computers [59,60]. The aim is to evolve a secure digital communication that remarkably pushes forward the limits of legacy digital watermarking schemes across all dimensions of performance metrics.…”

Section: Results Obtainedmentioning

confidence: 99%

A Computational Modelling and Algorithmic Design Approach of Digital Watermarking in Deep Neural Networks

Kavitha¹,

Eranna²,

Giriprasad³

2020

Adv. sci. technol. eng. syst. j.

View full text Add to dashboard Cite

In this paper we propose an algorithmic approach for Convolutional Neural Network (CNN) for digital watermarking which outperforms the existing frequency domain techniques in all aspects including security along with the criteria in the neural networks such as conditions embedded, and types of watermarking attack. This research addresses digital watermarking in deep neural networks and with comprehensive experiments through computational modeling and algorithm design, we examine the performance of the built system to demonstrate the potential of watermarking neural networks. The inability of intruder towards the retrieval of data without the knowledge of architecture and keys is also discussed and results of the proposed method are compared with the state of the art methods at different noises and attacks.

show abstract

“…Towards another approach with an effective teacher network, we perform online distillation (via collaborative learning) to transfer pose driven attention knowledge learned from VPN [30] to RGB stream. Distillation methods like [52], [53] are close to our approaches, however they are specific for image domain applications. In contrast, the extension of VPN: VPN++ is dedicated for combining cross-modal information pertaining to video domain applications.…”

Section: Related Workmentioning

confidence: 93%

VPN: Learning Video-Pose Embedding for Activities of Daily Living

Das

Sharma

Dai

et al. 2020

Lecture Notes in Computer Science

111

View full text Add to dashboard Cite

Many attempts have been made towards combining RGB and 3D poses for the recognition of Activities of Daily Living (ADL). ADL may look very similar and often necessitate to model fine-grained details to distinguish them. Because the recent 3D ConvNets are too rigid to capture the subtle visual patterns across an action, this research direction is dominated by methods combining RGB and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the absence of appropriate sensors. This limits the usage of aforementioned approaches in real-world applications requiring low latency. Then, how to best take advantage of 3D Poses for recognizing ADL? To this end, we propose an extension of a pose driven attention mechanism: Video-Pose Network (VPN), exploring two distinct directions. One is to transfer the Pose knowledge into RGB through a feature-level distillation and the other towards mimicking pose driven attention through an attention-level distillation. Finally, these two approaches are integrated into a single model, we call VPN++. We show that VPN++ is not only effective but also provides a high speed up and high resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the representative baselines on 4 public datasets. Code is available at https://github.com/srijandas07/vpnplusplus.

show abstract

Online Knowledge Distillation via Collaborative Learning

Cited by 247 publications

References 14 publications

Knowledge distillation in deep learning and its applications

Knowledge distillation in deep learning and its applications

A Computational Modelling and Algorithmic Design Approach of Digital Watermarking in Deep Neural Networks

VPN: Learning Video-Pose Embedding for Activities of Daily Living

Contact Info

Product

Resources

About