Pull &amp; Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

Zhou, Qihang; He, Shibo; Liu, Haoyu; Chen, Tao; Chen, Jiming

doi:10.1109/tcsvt.2022.3218587

Cited by 19 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Style distillation [32] aims to extract information on multiple styles from the teacher network's feature map, from which the student network can learn to improve its generalizability and reconstruct anomalous patterns. The pull and push [33] method addresses generalizability problems by maximizing pixel-wise discrepancies for anomalous regions and simultaneously minimizing discrepancies for normal regions between two networks.…”

Section: B Knowledge Distillation-based Anomaly Detectionmentioning

confidence: 99%

Anomaly Detection via a Bottleneck Structure Robust to Noise in In-Distribution Samples

Cho,

Lee,

Hyun

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Anomaly detection aims to distinguish data that are not part of in-distribution (ID) samples. It detects unknown defect images considered out-of-distribution (OOD) samples using only existing defect images in manufacturing. Recently, networks pretrained on large datasets, such as ImageNet, have been used to extract ID features. These methods use the distance between the extracted features of the testing sample and all the training samples as an anomaly score. This approach identifies anomaly samples simply and effectively without requiring additional training. However, the extracted feature is high dimensional; thus, it contains considerable unnecessary information, resulting in a sensitivity to noise in some ID samples. In addition, during the nearest-neighbor search process, false positive occurs as a result of an ID sample with a low probability of occurrence. This paper presents a feature extraction methodology that is robust to noise by adopting a bottleneck structure used in reverse knowledge distillation to solve the noise sensitivity problem. Next, the false positive was solved using K-means clustering-based augmentation for samples far from each cluster center. Finally, we propose a weighted scoring function that increases the anomaly score when the nearest ID samples from the testing sample are rare. Applying the proposed methodology to the electric vehicle battery dataset and CIFAR10 revealed that the performance improved compared to the baseline model.

show abstract

Section: B Knowledge Distillation-based Anomaly Detectionmentioning

confidence: 99%

Anomaly Detection via a Bottleneck Structure Robust to Noise in In-Distribution Samples

Cho,

Lee,

Hyun

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Momentum-based approaches have considerably enriched the field of selfsupervised learning with the teacher-student (TE-ST) formula. Prior works have carefully exploited the importance of TE-ST discrepancy, such as various knowledge distillation techniques in traditional supervised learning [15], [30]- [34] or semi-supervised learning [35], [36]. However, in self-supervised learning paradigm, this behavior, i.e.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Visual Representation Learning via Residual Momentum

Pham,

Niu,

Zhang

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Self-supervised learning (SSL) has emerged as a promising approach for learning representations from unlabeled data. Momentum-based contrastive frameworks such as MoCo-v3 have shown remarkable success among the many SSL methods proposed in recent years. However, a significant gap in encoder representation exists between the online encoder (student) and the momentum encoder (teacher) in these frameworks, limiting the performance on downstream tasks. We identify this gap as a bottleneck often overlooked in existing frameworks and propose "residual momentum" that explicitly reduces the gap during training to encourage the student to learn representations closer to the teacher's. We also reveal that a similar technique, knowledge distillation (KD), to reduce the distribution gap with cross-entropybased loss in supervised learning is useless in the SSL context and demonstrate that the intra-representation gap measured by cosine similarity is crucial for EMA-based SSLs. Extensive experiments on different benchmark datasets and architectures demonstrate the superiority of our method compared to state-of-theart contrastive learning baselines. Specifically, our method outperforms MoCo-v3 0.7% top-1 in ImageNet, 2.82% on CIFAR-100, 1.8% AP, and 3.0% AP75 on VOC detection pre-trained on the COCO dataset; it also improves DenseCL with 0.5% AP (800ep) and 0.6% AP75 (1600ep). Our work highlights the importance of reducing the teacher-student intra-gap in momentum-based contrastive learning frameworks and provides a practical solution for improving the quality of learned representations.

show abstract

MoEAD: A Parameter-Efficient Model for Multi-class Anomaly Detection

Meng,

Zhou

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Pull & Push: Leveraging Differential Knowledge Distillation for Efficient Unsupervised Anomaly Detection and Localization

Cited by 19 publications

References 39 publications

Anomaly Detection via a Bottleneck Structure Robust to Noise in In-Distribution Samples

Anomaly Detection via a Bottleneck Structure Robust to Noise in In-Distribution Samples

Self-Supervised Visual Representation Learning via Residual Momentum

MoEAD: A Parameter-Efficient Model for Multi-class Anomaly Detection

Contact Info

Product

Resources

About