CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation

Amara, Ibtihel; Ziaeefard, Maryam; Meyer, Brett H.; Gross, Warren J.; Clark, James J.

doi:10.1109/icpr56361.2022.9956061

Cited by 2 publications

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we first compared CSKD with some previous methods, including KD, 12 FitNet, 34 AT, 35 SP, 36 VID, 38 HKD, 39 MGD, 41 CRD, 40 virtual knowledge distillation (VKD), 42 curriculum expert selection for knowledge distillation (CESKD), 43 and multilevel attention-based sample correlations for knowledge distillation (MASCKD), 44 on benchmark datasets to verify its effectiveness. Then, we conducted representational transferability experiments to evaluate the quality of representations learned by the student network.…”

Section: Methodsmentioning

confidence: 99%

Cross-stage distillation with inverted bottleneck projectors

Sun,

Liu,

Zhang

et al. 2023

J. Electron. Imag.

View full text Add to dashboard Cite

Knowledge distillation, which transfers knowledge from a large model (the teacher) to a small model (the student), is a promising way in the field of lightweight model design. Existing distillation methods based on intermediate features mainly focus on knowledge transfer between the same stages of teacher and student networks, which may lead to student networks receiving semantically mismatched knowledge and missing contextual knowledge. To solve this problem, a cross-stage distillation with inverted bottleneck projectors is proposed. A cross-stage connection structure is designed that enables the student network to access the teacher network's most matched stages in semantics and further obtain rich contextual knowledge from multiple stages of the teacher. After establishing the cross-stage connection between the teacher network and the student network, inverted bottleneck projectors are employed to extract useful knowledge from multiple stages of the teacher. A stack structure with three-layer convolutions is proposed in every projector, which can make the knowledge of the teacher network more easily understood by the student network. The inverted bottleneck structure can reduce information loss, ensuring the integrity of knowledge transmission. Further, ReLU activations are incorporated into the projectors to remove features with lower response values, thereby filtering redundant elements. Extensive experiments on CIFAR100, ImageNet, Tiny-ImageNet, and STL-10 datasets demonstrate the effectiveness of the proposed approach.

show abstract