Fabrication and Characterization of Two-dimensional Fiber-optic Radiation Sensor for High Energy Photon Beam Therapy Dosimetry

Jang, Kyoung Won; Cho, Dong−Hyun; Shin, Sang‐Hun; Kim, Hyung-Shik; Yi, Jeong-Han; Lee, Bongsoo; Kim, Sin; Cho, Hyo-Sung

doi:10.3807/hkh.2007.18.4.241

Cited by 2 publications

(1 citation statement)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FitNet [38] pioneered feature distillation by utilizing the intermediate layers' features from the teacher model. To find the better feature layers for distillation, subsequent works [6,20] studied the factor of connection path cross multiple i ntermediate layers between teacher and student networks. Besides distilling the knowledge contained in samples, inter-samples relation, as structural information, was transferred to student models [33,34,43].…”

Section: Related Workmentioning

confidence: 99%

Generic-to-Specific Distillation of Masked Autoencoders

Huang¹,

ZhiLiang²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large vision Transformers (ViTs) driven by selfsupervised pre-training mechanisms achieved unprecedented progress. Lightweight ViT models limited by the model capacity, however, benefit little from those pretraining mechanisms. Knowledge distillation defines a paradigm to transfer representations from large (teacher) models to small (student) ones. However, the conventional single-stage distillation easily gets stuck on task-specific transfer, failing to retain the task-agnostic knowledge crucial for model generalization. In this study, we propose generic-to-specific distillation (G2SD), to tap the potential of small ViT models under the supervision of large models pre-trained by masked autoencoders. In generic distillation, decoder of the small model is encouraged to align feature predictions with hidden representations of the large model, so that task-agnostic knowledge can be transferred. In specific distillation, predictions of the small model are constrained to be consistent with those of the large model, to transfer task-specific features which guarantee task performance. With G2SD, the vanilla ViT-Small model respectively achieves 98.7%, 98.1% and 99.3% the performance of its teacher (ViT-Base) for image classification, object detection, and semantic segmentation, setting a solid baseline for two-stage vision distillation. Code will be available at https://github.com/pengzhiliang/G2SD.

show abstract