The Devil of Face Recognition Is in the Noise

Most teacher-student frameworks based on knowledge distillation (KD) depend on a strong congruent constraint on instance level. However, they usually ignore the correlation between multiple instances, which is also valuable for knowledge transfer. In this work, we propose a new framework named correlation congruence for knowledge distillation (CCKD), which transfers not only the instance-level information, but also the correlation between instances. Furthermore, a generalized kernel method based on Taylor series expansion is proposed to better capture the correlation between instances. Empirical experiments and ablation studies on image classification tasks (including CIFAR-100, ImageNet-1K) and metric learning tasks (including ReID and Face Recognition) show that the proposed CCKD substantially outperforms the original KD and achieves stateof-the-art accuracy compared with other SOTA KD-based methods. The CCKD can be easily deployed in the majority of the teacher-student framework such as KD and hintbased learning methods. Our code will be released, hoping to nourish our idea to other domains.

show abstract

“…In this experiment, instead of using KD loss, we adopt the L2-mimic loss. MS-Celeb-1M [9] and IMDB-Face [32] are used as training datasets.…”

Section: Face Recognition Results On Megafacementioning

confidence: 99%

“…Table 4: Results on Megaface. The teacher network is ResNet-50 trained on MsCeleb-1M [9] and IMDb-face [32] using Arc-Face [5]. The student network is MobileNetV2 with a width mul-tiplier=0.5.…”

Section: Face Recognition Results On Megafacementioning

confidence: 99%

Correlation Congruence for Knowledge Distillation

Peng¹,

Xiao

Li³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

468

246

View full text Add to dashboard Cite

show abstract

“…The presence of noisy training data may adversely affect the final performance of trained CNNs. Though a recent work [35] reports that deep CNNs still perform well even on noisy datasets containing sufficient clean data, this conclusion cannot be transferred to FR, and experiments demonstrate that noisy data apparently decrease the performance of the trained FR CNNs [42].…”

Section: Introductionmentioning

confidence: 94%

Noise-Tolerant Paradigm for Training Face Recognition CNNs

Huang²,

Zhang

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Benefit from large-scale training datasets, deep Convolutional Neural Networks(CNNs) have achieved impressive results in face recognition(FR). However, tremendous scale of datasets inevitably lead to noisy data, which obviously reduce the performance of the trained CNN models. Kicking out wrong labels from large-scale FR datasets is still very expensive, although some cleaning approaches are proposed. According to the analysis of the whole process of training CNN models supervised by angular margin based loss(AM-Loss) functions, we find that the θ distribution of training samples implicitly reflects their probability of being clean. Thus, we propose a novel training paradigm that employs the idea of weighting samples based on the above probability. Without any prior knowledge of noise, we can train high performance CNN models with largescale FR datasets. Experiments demonstrate the effectiveness of our training paradigm. The codes are available at https://github.com/huangyangyu/NoiseFace.

show abstract

“…We take two popular face recognition datasets MS-Celeb-1M [6] and IMDb-Face [28] as our training set and validate our method on MegaFace. The MS-Celeb-1M is a large public face dataset which contains one million identities with different age, sex, skin, and nationality and is widely used in face recognition area.…”

Section: Experiments On Face Recognitionmentioning

confidence: 99%

Knowledge Distillation via Route Constrained Optimization

Xiao

Peng

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

142

View full text Add to dashboard Cite

Distillation-based learning boost the performance of the miniaturized neural network based on the hypothesis that the representation of a teacher model can be used as structured and relatively weak supervision, and thus would be easily learned by a miniaturized model. However, we find that the representation of a converged heavy model is still a strong constraint for training a small student model, which leads to a high lower bound of congruence loss. In this work, inspired by [1] we consider the knowledge distillation from the perspective of curriculum learning by routing. Instead of supervising the student model with a converged teacher model, we supervised it with some anchor points selected from the route in parameter space that the teacher model passed by, as we called route constrained optimization (RCO). We experimentally demonstrate this simple operation greatly reduces the lower bound of congruence loss for knowledge distillation, hint and mimicking learning. On close-set classification tasks like CIFAR [16] and ImageNet [3], RCO improves knowledge distillation by 2.14% and 1.5% respectively. For the sake of evaluating the generalization, we also test RCO on the open-set face recognition task MegaFace. RCO achieves 84.3% accuracy on 1 vs. 1 million task with only 0.8 M parameters, which push the SOTA by a large margin.

show abstract

The Devil of Face Recognition Is in the Noise

Cited by 166 publications

References 26 publications

Correlation Congruence for Knowledge Distillation

Correlation Congruence for Knowledge Distillation

Noise-Tolerant Paradigm for Training Face Recognition CNNs

Knowledge Distillation via Route Constrained Optimization

Contact Info

Product

Resources

About