Focal and Global Knowledge Distillation for Detectors

Yang, Zhendong; Li, Zhe; Jiang, Xiaohu; Gong, Yuan; Yuan, Zehuan; Zhao, Danpei; Yuan, Chun

doi:10.48550/arxiv.2111.11837

Cited by 10 publications

(37 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early KD approaches for classification focus on transferring knowledge to student models by forcing their predictions to match those of the teacher [14]. More recent work [34,35,41] claims that feature imitation, i.e. forcing the intermediate feature maps of student models to match their teacher counterpart, is more effective for detection.…”

Section: Related Workmentioning

confidence: 99%

“…A vital challenge when performing feature imitation for dense object detectors is determining which feature regions to distil from the teacher model. Naively distilling all feature maps equally results in poor performance [10,30,35]. To solve this problem, FGFI [34] distils features that are covered by anchor boxes which have a high IoU with the GT.…”

Section: Related Workmentioning

confidence: 99%

“…To solve this problem, FGFI [34] distils features that are covered by anchor boxes which have a high IoU with the GT. However, distilling in this manner is still suboptimal [9,30,35,40,41]. TADF [30] suppresses foreground pixels according to a static 2D Gaussian fitted over the GT.…”

Section: Related Workmentioning

confidence: 99%

“…While soft label-based KD can be directly applied for classification, finding an equivalent for localisation remains a challenge. Recent work [9,10,30,34,35,37,41] alleviates this problem by forcing the student model to generate feature maps similar to the teacher counterpart; a process known as feature imitation.…”

Section: Introductionmentioning

confidence: 99%

“…This question is of the utmost importance for dense object detectors [7,16,19,31,38,42] because, unlike two-stage detectors [3,12,27], they do not use the RoIAlign [12] operation to explicitly pool and align object features; instead they output predictions at every location of the feature map [17]. Recent work [30,35] has shown that distilling the whole feature map with equal weighting is sub-optimal because not all features carry equally meaningful information. Therefore, a weighting mechanism that assigns appropriate importance to different regions, particularly to foreground regions near the objects, is highly desirable for dense object detectors, and has featured in recent work.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Prediction-Guided Distillation for Dense Object Detection

Yang¹,

Ochal²,

Storkey³

et al. 2022

Preprint

View full text Add to dashboard Cite

Real-world object detection models should be cheap and accurate. Knowledge distillation (KD) can boost the accuracy of a small, cheap detection model by leveraging useful information from a larger teacher model. However, a key challenge is identifying the most informative features produced by the teacher for distillation. In this work, we show that only a very small fraction of features within a groundtruth bounding box are responsible for a teacher's high detection performance. Based on this, we propose Prediction-Guided Distillation (PGD), which focuses distillation on these key predictive regions of the teacher and yields considerable gains in performance over many existing KD baselines. In addition, we propose an adaptive weighting scheme over the key regions to smooth out their influence and achieve even better performance. Our proposed approach outperforms current state-of-theart KD baselines on a variety of advanced one-stage detection architectures. Specifically, on the COCO dataset, our method achieves between +3.1% and +4.6% AP improvement using ResNet-101 and ResNet-50 as the teacher and student backbones, respectively. On the Crowd-Human dataset, we achieve +3.2% and +2.0% improvements in MR and AP, also using these backbones. Our code is available at https: //github.com/ChenhongyiYang/PGD.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Prediction-Guided Distillation for Dense Object Detection

Yang¹,

Ochal²,

Storkey³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Prediction-Guided Distillation for Dense Object Detection

Yang

Ochal

Storkey

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Real-world object detection models should be cheap and accurate. Knowledge distillation (KD) can boost the accuracy of a small, cheap detection model by leveraging useful information from a larger teacher model. However, a key challenge is identifying the most informative features produced by the teacher for distillation. In this work, we show that only a very small fraction of features within a groundtruth bounding box are responsible for a teacher's high detection performance. Based on this, we propose Prediction-Guided Distillation (PGD), which focuses distillation on these key predictive regions of the teacher and yields considerable gains in performance over many existing KD baselines. In addition, we propose an adaptive weighting scheme over the key regions to smooth out their influence and achieve even better performance. Our proposed approach outperforms current state-of-theart KD baselines on a variety of advanced one-stage detection architectures. Specifically, on the COCO dataset, our method achieves between +3.1% and +4.6% AP improvement using ResNet-101 and ResNet-50 as the teacher and student backbones, respectively. On the CrowdHuman dataset, we achieve +3.2% and +2.0% improvements in MR and AP, also using these backbones. Our code is available at https://github.com/ ChenhongyiYang/PGD.

show abstract