Long short-term memory language models with additive morphological features for automatic speech recognition

Renshaw, Daniel; Hall, Keith B.

doi:10.1109/icassp.2015.7178972

Cited by 7 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Theoretical analysis also helped us better understand the performance of the incremental algorithm. The natural future work is to extend our approach to other advanced NNLMs beyond CBOW and Skip-gram such as dependency RNN (Mirowski and Vlachos 2015) and LSTM or deeper RNN models (Renshaw and Hall 2015). 5 the LORELEI Contract HR0011-15-2-0025 with the US Defense Advanced Research Projects Agency (DARPA).…”

Section: Discussionmentioning

confidence: 99%

Incrementally Learning the Hierarchical Softmax Function for Neural Language Models

Peng

Song

et al. 2017

AAAI

View full text Add to dashboard Cite

Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors.

show abstract

Section: Discussionmentioning

confidence: 99%

Incrementally Learning the Hierarchical Softmax Function for Neural Language Models

Peng

Song

et al. 2017

AAAI

View full text Add to dashboard Cite

show abstract

“…In the former, both local and global features were examined in [9,10]. Here, the local features are related to morpheme [10,11,12] or word [9,13] and the global features are related to sentence [14] or document [3,10,15]. Further, sociosituational settings were also examined for adaptation in [9].…”

Section: Related Workmentioning

confidence: 99%

Use of Global and Acoustic Features Associated with Contextual Factors to Adapt Language Models for Spontaneous Speech Recognition

Toyama¹,

Saito

Minematsu

2017

Interspeech 2017

View full text Add to dashboard Cite

In this study, we propose a new method of adapting language models for speech recognition using para-linguistic and extralinguistic features in speech. When we talk with others, we often change the way of lexical choice and speaking style according to various contextual factors. This fact indicates that the performance of automatic speech recognition can be improved by taking the contextual factors into account, which can be estimated from speech acoustics. In this study, we attempt to find global and acoustic features that are associated with those contextual factors, then integrate those features into Recurrent Neural Network (RNN) language models for speech recognition. In experiments, using Japanese spontaneous speech corpora, we examine how i-vector and openSMILE are associated with contextual factors. Then, we use those features in the reranking process of RNN-based language models. Results show that perplexity is reduced by 16% relative and word error rate is reduced by 2.1% relative for highly emotional speech.

show abstract

“…Another way of integrating word compositions into CSLMs is to follow an additive approach, such that, a word can be represented as the sum of vectors representing its morphemes. This approach was explored in LSTM NNLMs trained for a word vocabulary and and speech recognition accuracy was improved for highly inflectional Russian language [16].…”

Section: Introductionmentioning

confidence: 99%

Compositional Neural Network Language Models for Agglutinative Languages

Arisoy¹,

Saraçlar²

2016

Interspeech 2016

View full text Add to dashboard Cite

Continuous space language models (CSLMs) have been proven to be successful in speech recognition. With proper training of the word embeddings, words that are semantically or syntactically related are expected to be mapped to nearby locations in the continuous space. In agglutinative languages, words are made up of concatenation of stems and suffixes and, as a result, compositional modeling is important. However, when trained on word tokens, CSLMs do not explicitly consider this structure. In this paper, we explore compositional modeling of stems and suffixes in a long short-term memory neural network language model. Our proposed models jointly learn distributed representations for stems and endings (concatenation of suffixes) and predict the probability for stem and ending sequences. Experiments on the Turkish Broadcast news transcription task show that further gains on top of a state-of-theart stem-ending-based n-gram language model can be obtained with the proposed models.

show abstract

Long short-term memory language models with additive morphological features for automatic speech recognition

Cited by 7 publications

References 11 publications

Incrementally Learning the Hierarchical Softmax Function for Neural Language Models

Incrementally Learning the Hierarchical Softmax Function for Neural Language Models

Use of Global and Acoustic Features Associated with Contextual Factors to Adapt Language Models for Spontaneous Speech Recognition

Compositional Neural Network Language Models for Agglutinative Languages

Contact Info

Product

Resources

About