DNN speaker adaptation using parameterised sigmoid and ReLU hidden activation functions

Zhang, C.; Woodland, Philip C.

doi:10.1109/icassp.2016.7472689

Cited by 56 publications

(37 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The weighted sum is added with a bias to determine whether this neuron should be activated. Commonly used activation functions include linear function, sigmoid function, exponential linear unit (ELU), and rectified linear unit (ReLU) . In computational materials science, the feature values are obtained by numerical simulations, corresponding to one or more target values such as potential energy or mechanical properties.…”

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

Machine learning and artificial neural network accelerated computational discoveries in materials science

Yang

Hou

Jiang

et al. 2019

WIREs Comput Mol Sci

View full text Add to dashboard Cite

Artificial intelligence (AI) has been referred to as the "fourth paradigm of science," and as part of a coherent toolbox of data-driven approaches, machine learning (ML) dramatically accelerates the computational discoveries. As the machinery for ML algorithms matures, significant advances have been made not only by the mainstream AI researchers, but also those work in computational materials science. The number of ML and artificial neural network (ANN) applications in the computational materials science is growing at an astounding rate. This perspective briefly reviews the state-of-the-art progress in some supervised and unsupervised methods with their respective applications. The characteristics of primary ML and ANN algorithms are first described. Then, the most critical applications of AI in computational materials science such as empirical interatomic potential development, ML-based potential, property predictions, and molecular discoveries using generative adversarial networks (GAN) are comprehensively reviewed. The central ideas underlying these ML applications are discussed, and future directions for integrating ML with computational materials science are given. Finally, a discussion on the applicability and limitations of current ML techniques and the remaining challenges are summarized.

show abstract

Section: Approaches In Computational Materials Sciencementioning

confidence: 99%

Machine learning and artificial neural network accelerated computational discoveries in materials science

Yang

Hou

Jiang

et al. 2019

WIREs Comput Mol Sci

View full text Add to dashboard Cite

show abstract

“…Here η and γ still refer to the output and input value scaling vectors for tanhη,γ (·) and ReLUη(·). Other types of parameterised activation functions have also been investigated for both conventional modelling [22][23][24][25] and speaker adaptation [26][27][28][29].…”

Section: Parameterised Activation Function For Stusmentioning

confidence: 99%

Semi-tied Units for Efficient Gating in LSTM and Highway Networks

Zhang

Woodland

2018

Interspeech 2018

View full text Add to dashboard Cite

Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a separate full weight matrix. This issue can be severe since several gates can be used together in e.g. an LSTM cell. This paper proposes a semi-tied unit (STU) approach to solve this efficiency issue, which uses one shared weight matrix to replace those in all the units in the same layer. The approach is termed "semi-tied" since extra parameters are used to separately scale each of the shared output values. These extra scaling factors are associated with the network activation functions and result in the use of parameterised sigmoid, hyperbolic tangent, and rectified linear unit functions. Speech recognition experiments using British English multi-genre broadcast data showed that using STUs can reduce the calculation and storage cost by a factor of three for highway networks and four for LSTMs, while giving similar word error rates to the original models.

show abstract

“…Alternatively, the scaling values can also be linear as in [11], but this will not be investigated in the paper. Subsequently, the lth hidden layer activation outputs are computed by…”

Section: Feature Based Hidden Output Scalingmentioning

confidence: 99%

“…Model based adaptation approaches adapt models containing acoustic condition dependent parameters. These parameters can be linear transformation on the Gaussian mixture model (GMM) parameters like maximum likelihood linear regression (MLLR) transformation [3,4,5], weight vector for GMM mean or DNN transformation interpolation [6,7,8,9], or scaling values on the DNN hidden outputs [10,11]. The adaptation can be applied on evaluation data directly, or after the adaptive training [12] which jointly trains both speaker dependent and independent parameters on training data.…”

Section: Introductionmentioning

confidence: 99%

RNN-LDA Clustering for Feature Based DNN Adaptation

Xie¹,

Liu²,

Lee³

et al. 2017

Interspeech 2017

View full text Add to dashboard Cite

Model based deep neural network (DNN) adaptation approaches often require multi-pass decoding in test time. Input feature based DNN adaptation, for example, based on latent Dirichlet allocation (LDA) clustering, provide a more efficient alternative. In conventional LDA clustering, the transition and correlation between neighboring clusters is ignored. In order to address this issue, a recurrent neural network (RNN) based clustering scheme is proposed to learn both the standard LDA cluster labels and their natural correlation over time in this paper. In addition to directly using the resulting RNN-LDA as input features during DNN adaptation, a range of techniques were investigated to condition the DNN hidden layer parameters or activation outputs on the RNN-LDA features. On a DARPA Gale Mandarin Chinese broadcast speech transcription task, the proposed RNN-LDA cluster features adapted DNN system outperformed both the baseline un-adapted DNN system and conventional LDA features adapted DNN system by 8% relative on the most difficult Phoenix TV subset. Consistent improvements were also obtained after further combination with model based adaptation approaches.

show abstract

DNN speaker adaptation using parameterised sigmoid and ReLU hidden activation functions

Cited by 56 publications

References 20 publications

Machine learning and artificial neural network accelerated computational discoveries in materials science

Machine learning and artificial neural network accelerated computational discoveries in materials science

Semi-tied Units for Efficient Gating in LSTM and Highway Networks

RNN-LDA Clustering for Feature Based DNN Adaptation

Contact Info

Product

Resources

About