Self-Distillation from the Last Mini-Batch for Consistency Regularization

Shen, Yiqing; Xu, Liwu; Yang, Yuzhe; Li, Yaqian; Guo, Yandong

doi:10.1109/cvpr52688.2022.01164

Cited by 51 publications

(24 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The label refurbishment in Section 3.4.2 only utilizes discrete hard labels, overlooking the rich information in the continuous soft distributions. Here, we record the exponential moving average (EMA) of historical logits to impose temporal consistency regularization [ 32 , 33 ].…”

Section: Our Methodsmentioning

confidence: 99%

“…As for theoretical analysis, there are various opinions that include label smoothing regularization [ 29 ], the multi-view hypothesis [ 30 ], and loss landscape flattening [ 31 ]. Similarly to our method, PS-KD [ 32 ] trained a model with soft targets, which were a weighted summation of the hard targets and the last-epoch predictions, and DLB [ 33 ] used predictions from the last iteration as soft targets. However, we considered the entire prediction history and maintained an exponential moving average of the predictions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Point Cloud Instance Segmentation with Inaccurate Bounding-Box Annotations

Peng

Hong

Chen

et al. 2023

Sensors

View full text Add to dashboard Cite

Most existing point cloud instance segmentation methods require accurate and dense point-level annotations, which are extremely laborious to collect. While incomplete and inexact supervision has been exploited to reduce labeling efforts, inaccurate supervision remains under-explored. This kind of supervision is almost inevitable in practice, especially in complex 3D point clouds, and it severely degrades the generalization performance of deep networks. To this end, we propose the first weakly supervised point cloud instance segmentation framework with inaccurate box-level labels. A novel self-distillation architecture is presented to boost the generalization ability while leveraging the cheap but noisy bounding-box annotations. Specifically, we employ consistency regularization to distill self-knowledge from data perturbation and historical predictions, which prevents the deep network from overfitting the noisy labels. Moreover, we progressively select reliable samples and correct their labels based on the historical consistency. Extensive experiments on the ScanNet-v2 dataset were used to validate the effectiveness and robustness of our method in dealing with inexact and inaccurate annotations.

show abstract

Section: Our Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Point Cloud Instance Segmentation with Inaccurate Bounding-Box Annotations

Peng

Hong

Chen

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…Additionally, α represents the momentum term, which determines the proportion of historical prediction participating in the t-th iteration. Shen et al [138] use half of each mini-batch for extracting smoothed labels from previous iterations and the other half for providing soft targets for self-regularization.…”

Section: Aspect Of Historical Typementioning

confidence: 99%

“…(4) Probability predictions. The recorded probability predictions of specific instances are adopted for ensembling [132]- [135] and self-teaching [136]- [138]. (5) Loss values.…”

Section: Introductionmentioning

confidence: 99%

A Survey of Historical Learning: Learning Models with Learning History

Li¹,

Ge²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

New knowledge originates from the old. The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models. In this survey, we comprehensively review and summarize the topic-"Historical Learning: Learning Models with Learning History", which learns better neural models with the help of their learning history during its optimization, from three detailed aspects: Historical Type (what), Functional Part (where) and Storage Form (how). To our best knowledge, it is the first survey that systematically studies the methodologies which make use of various historical statistics when training deep neural networks. The discussions with related topics like recurrent/memory networks, ensemble learning, and reinforcement learning are demonstrated. We also expose future challenges of this topic and encourage the community to pay attention to the think of historical learning principles when designing algorithms. The paper list related to historical learning is available at https://github.com/Martinser/Awesome-Historical-Learning.

show abstract

“…Recently, there has been many knowledge distillation mechanisms (Gou et al, 2021) that vary in either the distillation mechanisms (e.g., offline distillation, online distillation, and self-distillation) or the types of knowledge distilled (e.g., logits, feature knowledge, and relational knowledge). Specifically, offline distillation utilises a two-stage approach that requires the teacher model to be pre-trained as a prior and fixed for knowledge transfer (Liu et al, 2021;Zhao et al, 2022;Mirzadeh et al, 2020;Wu et al, 2020;, while online and self-distillation instead argues that a pre-trained large teacher model is not always available for the tasks of interest, thus propose to update both the teacher model and the student model simultaneously in an end-toend manner (Yuan et al, 2020;Mobahi et al, 2020;Zhao et al, 2021;Xu et al, 2022;Shen et al, 2022;Gou et al, 2022;Li et al, 2018;Wu & Gong, 2021;Zhang et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

Multi-target Knowledge Distillation via Student Self-reflection

Gou

Xiong

et al. 2023

Int J Comput Vis

View full text Add to dashboard Cite

Knowledge distillation is a simple yet effective technique for deep model compression, which aims to transfer the knowledge learned by a large teacher model to a small student model. To mimic how the teacher teaches the student, existing knowledge distillation methods mainly adapt an unidirectional knowledge transfer, where the knowledge extracted from different intermedicate layers of the teacher model is used to guide the student model. However, it turns out that the students can learn more effectively through multi-stage learning with a self-reflection in the real-world education scenario, which is nevertheless ignored by current knowledge distillation methods. Inspired by this, we devise a new knowledge distillation framework entitled multi-target knowledge distillation via student self-reflection or MTKD-SSR, which can not only enhance the teacher’s ability in unfolding the knowledge to be distilled, but also improve the student’s capacity of digesting the knowledge. Specifically, the proposed framework consists of three target knowledge distillation mechanisms: a stage-wise channel distillation (SCD), a stage-wise response distillation (SRD), and a cross-stage review distillation (CRD), where SCD and SRD transfer feature-based knowledge (i.e., channel features) and response-based knowledge (i.e., logits) at different stages, respectively; and CRD encourages the student model to conduct self-reflective learning after each stage by a self-distillation of the response-based knowledge. Experimental results on five popular visual recognition datasets, CIFAR-100, Market-1501, CUB200-2011, ImageNet, and Pascal VOC, demonstrate that the proposed framework significantly outperforms recent state-of-the-art knowledge distillation methods.

show abstract

Self-Distillation from the Last Mini-Batch for Consistency Regularization

Cited by 51 publications

References 25 publications

Point Cloud Instance Segmentation with Inaccurate Bounding-Box Annotations

Point Cloud Instance Segmentation with Inaccurate Bounding-Box Annotations

A Survey of Historical Learning: Learning Models with Learning History

Multi-target Knowledge Distillation via Student Self-reflection

Contact Info

Product

Resources

About