Anomaly detection based on multi-teacher knowledge distillation

Ma, Ye; Xu, Jiang; Guan, Nan; Wang, Yi

doi:10.1016/j.sysarc.2023.102861

Cited by 13 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The intuition of KD is to minimize the divergence between logits outputs from teacher model and student model. Therefore, student model can learn the knowledge from teacher model and acquire a better performance because student model could approach output from teacher model 30 , 31 . The loss of knowledge distillation could be computed by Kullback-Leiler divergence: Based on the type of transferred knowledge, knowledge distillation can be divided into three categories: (1) output transfer; (2) feature transfer; (3) relation transfer.…”

Section: Background and Related Workmentioning

confidence: 99%

Global prototype distillation for heterogeneous federated learning

Wu,

Chen,

Nie

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Federated learning is a distributed machine learning paradigm where the goal is to collaboratively train a high quality global model while private training data remains local over distributed clients. However, heterogenous data distribution over clients is severely challenging for federated learning system, which severely damage the quality of model. In order to address this challenge, we propose global prototype distillation (FedGPD) for heterogenous federated learning to improve performance of global model. The intuition is to use global class prototypes as knowledge to instruct local training on client side. Eventually, local objectives will be consistent with the global optima so that FedGPD learns an improved global model. Experiments show that FedGPD outperforms previous state-of-art methods by 0.22% ~1.28% in terms of average accuracy on representative benchmark datasets.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Global prototype distillation for heterogeneous federated learning

Wu,

Chen,

Nie

et al. 2024

Sci Rep

View full text Add to dashboard Cite

show abstract

Section: Knowledge Distillationmentioning

confidence: 99%

FedGPD: Global Prototype Distillation InHeterogeneous Federated Learning

Wu,

Chen,

Nie

et al. 2024

Preprint

View full text Add to dashboard Cite

Federated learning is a distributed machine learning paradigm where the goal is to collaboratively train a high quality globalmodel while private training data remains local over distributed clients. However, heterogenous data distribution over clientsis severely challenging for federated learning system, which severely damage the quality of model. In order to address thischallenge, global prototype distillation (FedGPD) for heterogenous federated learning is proposed to improve performance ofglobal model. The intuition is to use global class prototypes as knowledge to instruct local training on client side. Eventually,local objectives will be consistent with the global optima so that FedGPD learns an improved global model. Experiments showthat FedGPD outperforms previous state-of-art methods at the cost of a slight communication overhead.

show abstract