Student-Teacher Learning from Clean Inputs to Noisy Inputs

Hong, Guanzhe; Mao, Zhiyuan; Lin, Xiaojun; Chan, Stanley H.

doi:10.1109/cvpr46437.2021.01190

Cited by 7 publications

(2 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has also been shown to work for distilling knowledge of non-neural network machine-learning models [11]. In [8], [12] KD was used to transfer knowledge across different types of sensors for either image reconstruction or high-level tasks, and in [17] it was theoretically analyzed. In this work, we use KD to distill not just a deep model, but also the non-learned (manually engineered) algorithms in the ISP and the information gained in the physical process of acquiring a better signal (that has a better SNR).…”

Section: Isp For Visionmentioning

confidence: 99%

ISP Distillation

Schwartz

Bronstein

Giryes

2023

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

Nowadays, many of the images captured are 'observed' by machines only and not by humans, e.g., in autonomous systems. High-level machine vision models, such as object recognition or semantic segmentation, assume images are transformed into some canonical image space by the camera Image Signal Processor (ISP). However, the camera ISP is optimized for producing visually pleasing images for human observers and not for machines. Therefore, one may spare the ISP compute time and apply vision models directly to RAW images. Yet, it has been shown that training such models directly on RAW images results in a performance drop. To mitigate this drop, we use a RAW and RGB image pairs dataset, which can be easily acquired with no human labeling. We then train a model that is applied directly to the RAW data by using knowledge distillation such that the model predictions for RAW images will be aligned with the predictions of an off-the-shelf pre-trained model for processed RGB images. Our experiments show that our performance on RAW images for object classification and semantic segmentation is significantly better than models trained on labeled RAW images. It also reasonably matches the predictions of a pre-trained model on processed RGB images, while saving the ISP compute overhead.

show abstract

Section: Isp For Visionmentioning

confidence: 99%

ISP Distillation

Schwartz

Bronstein

Giryes

2023

IEEE Open J. Signal Process.

View full text Add to dashboard Cite

show abstract

“…BiT [1] and BERT [2]); 2) neural networks architecture searching (NAS), which designs or searches an optimal architecture for various downstream tasks (e.g. EfficientNet [3] and NASNet [4]); 3) noise learning, which introduces noises into original inputs to help neural networks learning a generalized representation capability on noisy inputs (Noisy Student-Teacher Learning [5]), adversarial noise (FreeLB [6]) etc. However, these methods either require additional data like transfer learning or incur huge computational cost in training like NAS and noisy learning.…”

Section: Introductionmentioning

confidence: 99%

Enhanced gradient learning for deep neural networks

Yan

Yang

Chen

et al. 2021

IET Image Processing

View full text Add to dashboard Cite

Deep neural networks have achieved great success in both computer vision and natural language processing tasks. How to improve the gradient flows is crucial in training very deep neural networks. To address this challenge, a gradient enhancement approach is proposed through constructing the short circuit neural connections. The proposed short circuit is a unidirectional neural connection that back propagates the sensitivities rather than gradients in neural networks from the deep layers to the shallow layers. Moreover, the short circuit is further formulated as a gradient truncation operation in its connecting layers, which can be plugged into the backbone models without introducing extra training parameters. Extensive experiments demonstrate that the deep neural networks, with the help of short circuit connection, gain a large margin of improvement over the baselines on both computer vision and natural language processing tasks. The work provides the promising solution to the low-resource scenarios, such as, intelligence transport systems of computer vision, question answering of natural language processing.

show abstract

Adaptive Selective Knowledge Distillation: Not Blindly Accepting Teachers as Oracles

Peng,

Zhang,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Student-Teacher Learning from Clean Inputs to Noisy Inputs

Cited by 7 publications

References 13 publications

ISP Distillation

ISP Distillation

Enhanced gradient learning for deep neural networks

Adaptive Selective Knowledge Distillation: Not Blindly Accepting Teachers as Oracles

Contact Info

Product

Resources

About