Feature Normalized Knowledge Distillation for Image Classification

Xu, Kunran; Lai, Rui; Li, Yishi; Gu, Lin

doi:10.1007/978-3-030-58595-2_40

Cited by 53 publications

(26 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To deploy deep HAR model on mobile wearable devices, model compression with small accuracy diminution is a critical and challenging mission. Universally, model compression methods include pruning [32], [33], [34], [35], [36], quantization [37], [38], low-rank approximation & sparsity [39], [40], and knowledge distillation [27], [41], [42]. Model pruning aims to prune non-significant weights in large models, and those pruned large-sparse models also have significant performance [43].…”

Section: Related Workmentioning

confidence: 99%

“…Knowledge distilling [27], [41], [42] plays an vital role in developing deep learning models that are friendly to wearable embedded devices. According to the distilled elements in the deep models [25], we can divide KD into response-based KD [27], [42], feature-based KD [41], [41], [49], and relation-based KD [50], [51]. Hinton et al [27] have formally popularized the idea of knowledge distillation in 2015.…”

Section: Related Workmentioning

confidence: 99%

“…Neurons or features of intermediate layers of deep models are another forms of knowledge to distill. Xu et al [41] analyzed the relationship between logits distillation and label smoothing. They use a normalized feature distilling to improve the temperature mechanism of the vanilla knowledge distillation.…”

Section: Related Workmentioning

confidence: 99%

“…After feature distilling in the early stages, both the manually labeled hard targets and soft targets provided by the teacher model in the final stage contrasted the whole student model's response logits. The hard one-hot labels are not always ground-truth in the real situation [41]. Manually labeled hard labels may introduce new noise for the HAR tasks.…”

Section: Logits Distillationmentioning

confidence: 99%

See 3 more Smart Citations

A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation

Chen¹,

Luo²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

Human activity recognition (HAR) based on IMU sensors is an essential domain in ubiquitous computing. Because of the improving trend to deploy artificial intelligence into IoT devices or smartphones, more researchers design the HAR models for embedded devices. We propose a plug-and-play HAR modeling pipeline with multi-level distillation to build deep convolutional HAR models with native support of embedded devices. SMLDist consists of stage distillation, memory distillation, and logits distillation, which covers all the information flow of the deep models. Stage distillation constrains the learning direction of the intermediate features.Memory distillation teaches the student models how to explain and store the inner relationship between high-dimensional features based on Hopfield networks. Logits distillation constructs distilled logits by a smoothed conditional rule to keep the probable distribution and improve the correctness of the soft target. We compare the performance of accuracy, F1 macro score, and energy cost on the embedded platform of various state-of-the-art HAR frameworks with a MobileNet V3 model built by SMLDist. The produced model has well balance with robustness, efficiency, and accuracy. SMLDist can also compress the models with minor performance loss in an equal compression rate than other state-of-the-art knowledge distillation methods on seven public datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Logits Distillationmentioning

confidence: 99%

See 2 more Smart Citations

A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation

Chen¹,

Luo²,

Zhao³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Generally, most of KD methods distill the response-based knowledge (e.g., soft logits of the output layer) from a large teacher network and transfer it to a small student [20,32,22]. To overcome the limitation of knowledge from the output layer of teacher, the featurebased knowledge from the middle layers of teacher is also used to train the student [21,18,19]. Unlike both the response-based and the feature-based knowledge from individual instances, the relation-based knowledge from instance relations is modelled for improving student learning [37,38,15,16,17].…”

Section: Introductionmentioning

confidence: 99%

Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Sun¹,

Gou²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Knowledge distillation (KD), as an efficient and effective model compression technique, has been receiving considerable attention in deep learning. The key to its success is to transfer knowledge from a large teacher network to a small student one. However, most of the existing knowledge distillation methods consider only one type of knowledge learned from either instance features or instance relations via a specific distillation strategy in teacher-student learning. There are few works that explore the idea of transferring different types of knowledge with different distillation strategies in a unified framework. Moreover, the frequently used offline distillation suffers from a limited learning capacity due to the fixed teacher-student architecture. In this paper we propose a collaborative teacher-student learning via multiple knowledge transfer (CTSL-MKT) that prompts both self-learning and collaborative learning. It allows multiple students learn knowledge from both individual instances and instance relations in a collaborative way. While learning from themselves with self-distillation, they can also guide each other via online distillation. The experiments and ablation studies on four image datasets demonstrate that the proposed CTSL-MKT significantly outperforms the state-of-the-art KD methods.

show abstract