P2SGrad: Refined Gradients for Optimizing Deep Face Models

Abstract

Cosine-based softmax losses [20,29,27,3] significantly improve the performance of deep face recognition networks. However, these losses always include sensitive hyper-parameters which can make training process unstable, and it is very tricky to set suitable hyper parameters for a specific dataset. This paper addresses this challenge by directly designing the gradients for adaptively training deep neural networks. We first investigate and unify previous cosine softmax losses by analyzing their gradients. This unified view inspires us to propose a novel gradient called P2SGrad (Probability-to-Similarity Gradient), which leverages a cosine similarity instead of classification probability to directly update the testing metrics for updating neural network parameters. P2SGrad is adaptive and hyperparameter free, which makes the training process more efficient and faster. We evaluate our P2SGrad on three face recognition benchmarks, LFW [7], MegaFace [8], and IJB-C [16]. The results show that P2SGrad is stable in training, robust to noise, and achieves state-of-the-art performance on all the three benchmarks.

A Comparative Study on Recent Neural Spoofing Countermeasures for Synthetic Speech Detection

“…This loss is a mean-square error (MSE) between a network's output and a scalar target value, but the network's output here is a cosine distance cos θ j,k rather than an unconstrained value. The gradient ∂L (p2s) /∂oj can be shown to be identical to the probability-to-similarity gradient (P2SGrad) [8], and this points to the same optimal direction as the margin-based softmax. We refer to the new loss function as MSE for P2SGrad.…”

“…The discriminative power of the margin-based softmax has been testified in a few speech [27] and image processing tasks [3], but its performance is sensitive to the hyper-parameter setting [8].…”

“…We refer to the new loss function as MSE for P2SGrad. Note that the P2SGrad paper defines only the gradient but not the loss function [8]. During inference, we may set s = cos θj,1 as the output score if k = 1 indicates the bonafide class.…”

“…Results on loss functions demonstrate the decent performance of a simple sigmoid function, which makes the gain of the margin-based softmax trivial. The new and simple loss function based on P2SGrad [8] is a competitive alternative. By combining the new loss function and an LCNN with average pooling, one of the CM achieved an equal error rate (EER) of 1.92% in the best training-evaluation round, which is the lowest for a single CM on the database without data augmentation as far as we know.…”

Deep Face Recognition Model Compression via Knowledge Transfer and Distillation

“…We further incorporated modifications to above architecture pertaining to face recognition domain. We set 112 × 112 image resolution for teacher network whereas different resolutions of 96 × 96, IJB-C 1:1 TAR (in %)@FAR Method 10 −4 10 −5 Crystal Loss [25] 92.50 87.75 P2SGrad [31] 92.25 87.84 Fixed AdaCos [30] 92.35 87.87 Dynamic AdaCos [30] 92.40 88.03 ArcFace [8] 95.65 93.15 Ours 96.39 94.20…”