On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification

Abstract

Deep representation learning has gained significant momentum in advancing text-dependent speaker verification (TD-SV) systems. When designing deep neural networks (DNN) for extracting bottleneck (BN) features, the key considerations include training targets, activation functions, and loss functions. In this paper, we systematically study the impact of these choices on the performance of TD-SV. For training targets, we consider speaker identity, time-contrastive learning (TCL), and auto-regressive prediction coding, with the first being supervised and the last two being self-supervised. Furthermore, we study a range of loss functions when speaker identity is used as the training target. With regard to activation functions, we study the widely used sigmoid function, rectified linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally show that GELU is able to reduce the error rates of TD-SV significantly compared to sigmoid, irrespective of the training target. Among the three training targets, TCL performs the best. Among the various loss functions, cross-entropy, joint-softmax, and focal loss functions outperform the others. Finally, the score-level fusion of different systems is also able to reduce the error rates. To evaluate the representation learning methods, experiments are conducted on the RedDots 2016 challenge database consisting of short utterances for TD-SV systems based on classic Gaussian mixture model-universal background model (GMM-UBM) and i-vector methods.

Camellia oleifera Fruit Detection Algorithm in Natural Environment Based on Lightweight Convolutional Neural Network

“…The activation function is a crucial part of a neural network, responsible for adding nonlinear factors to the output of neurons in the previous layer so that the network model can fit nonlinear functions, thus improving the network model characterization ability. In order to alleviate the accuracy degradation of the backbone network after lightweight improvement and to improve the model generalization ability and detection accuracy, the Gaussian Error Linear Units (GELU) activation function is introduced in this paper [35,36]. This function introduces the idea of stochastic regularity, a stochastic regular transformation method, which is essentially a fusion of Dropout, Zoneout, and ReLU, as shown in Equation (1).…”