Neural Network Language Modeling with Letter-Based Features and Importance Sampling

Xu, Hainan; Li, Ké; Wang, Yiming; Wang, Jian; Kang, Shiyin; Xie, Chen; Povey, Daniel; Khudanpur, Sanjeev

doi:10.1109/icassp.2018.8461704

Cited by 54 publications

(48 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Their application varies from identifying patterns in text [42] to data extraction [43], automatic speech recognition, machine translation, and spell checking [44,45]. Neural Network Language models offer an improved version [46], both having the potential to be integrated into computer-assisted tools for supporting text reviewers.…”

Section: Natural Language Processing Approachesmentioning

confidence: 99%

Hit or Miss? Evaluating the Potential of a Research Niche: A Case Study in the Field of Virtual Quality Management

et al. 2019

View full text Add to dashboard Cite

When knowledge is developed fast, as it is the case so often nowadays, one of the main difficulties in initiating new research in any field is to identify the domain's specific state-of-the-art and trends. In this context, to evaluate the potential of a research niche by assisting the literature review process and to add a new and modern large-scale and automated dimension to it, the paper proposes a methodology that uses "Latent Semantic Analysis" (LSA) for identifying trends, focused within the knowledge space created at the intersection of three sustainability-related methodologies/concepts: "virtual Quality Management" (vQM), "Industry 4.0", and "Product Life-Cycle" (PLC). The LSA was applied to a significant number of scientific papers published around these concepts to generate ontology charts that describe the knowledge structure of each by the frequency, position, and causal relation of associated notions. These notions are combined for defining the common high-density knowledge zone from where new technological solutions are expected to emerge throughout the PLC. The authors propose the concept of the knowledge space, which is characterized through specific descriptors with their own evaluation scales, obtained by processing the emerging information as identified by a combination of classic and innovative techniques. The results are validated through an investigation that surveys a relevant number of general managers, specialists, and consultants in the field of quality in the automotive sector from Romania. This practical demonstration follows each step of the theoretical approach and yields results that prove the capability of the method to contribute to the understanding and elucidation of the scientific area to which it is applied. Once validated, the method could be transferred to fields with similar characteristics. Even if their creators endowed them with a clear meaning at an incipient stage, when they become more popular in an emerging area, these concepts are quickly surrounded by a large amount of new knowledge that is developed with an amazing speed, enriching and enlarging their initial sphere.The "virtual Quality Management" (vQM) concept could be a significant example for the circumstances described previously. It is born through a semantic operation, joining two established and mature concepts: "virtual" and "QM", thus it is representative for an area which is in a period of high dynamic development and of interest for companies preoccupied with sustainability from the perspective of operations management and organizational culture.In this context in which the amount of information relating to new concepts quickly reaches unmanageable levels, regardless of the field, solutions that can analyze extended documentation with the purpose of disambiguating information and capturing the essentials, thus creating knowledge, become the focus of attention and gain in importance. Traditional solutions for that purpose lay in the literature review process, trying to collect, select, filter, and struc...

show abstract

Section: Natural Language Processing Approachesmentioning

confidence: 99%

Hit or Miss? Evaluating the Potential of a Research Niche: A Case Study in the Field of Virtual Quality Management

et al. 2019

View full text Add to dashboard Cite

show abstract

“…In order to improve the 1-best recognition hypotheses, we explored 3 types of NNLMs. Along with recurrent neural network language model (RNN-LM) integrated with the Kaldi toolkit [39], we also investigated Transformer-XL [40] and FRequency AGnostic word Embedding (FRAGE) with ASGD Weight-Dropped (AWD) Long Short-Term Memory (LSTM) Mixture of Softmaxes (MoS) [41], which are the current stateof-the-art for large and medium size vocabulary language modeling tasks respectively. Initial RNN-LM, Transformer-XL, and FRAGE with AWD-LSTM-MoS were trained on word-level training set transcriptions.…”

Section: Final Acoustic Modelsmentioning

confidence: 99%

The STC ASR System for the VOiCES from a Distance Challenge 2019

Medennikov

Khokhlov²,

Romanenko

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

This paper is a description of the Speech Technology Center (STC) automatic speech recognition (ASR) system for the "VOiCES from a Distance Challenge 2019". We participated in the Fixed condition of the ASR task, which means that the only training data available was an 80-hour subset of the Lib-riSpeech corpus. The main difficulty of the challenge is a mismatch between clean training data and distant noisy development/evaluation data. In order to tackle this, we applied room acoustics simulation and weighted prediction error (WPE) dereverberation. We also utilized well-known speaker adaptation using x-vector speaker embeddings, as well as novel room acoustics adaptation with R-vector room impulse response (RIR) embeddings. The system used a lattice-level combination of 6 acoustic models based on different pronunciation dictionaries and input features. N-best hypotheses were rescored with 3 neural network language models (NNLMs) trained on both words and sub-word units. NNLMs were also explored for out-of-vocabulary (OOV) words handling by means of artificial texts generation. The final system achieved Word Error Rate (WER) of 14.7% on the evaluation data, which is the best result in the challenge.

show abstract

“…Compared with PLSA and LDA, which describes the "word-document" co-occurrence, WVM attempts to discover the "word-word" co-occurrence dependence by latent topics. With the popularity of neural networks, neural language modeling such as recurrent neural network language model (RNNLM) was proposed in recent works [11]. Li et al [12] further proposed two adaptation models (a cache model and a DNN-based model) for RNNLM to capture the topic information and the long-distance triggers in ASR.…”

Section: Topic-based Language Model Adaptationmentioning

confidence: 99%

Topic-Aware Dialogue Speech Recognition with Transfer Learning

Song

Jiang

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

Dialogue speech widely exists in scenarios such as chitchat, meeting and customer service. General-purpose speech recognition systems usually neglect the topic information in the context of dialogue speech, which has great potential for improving the performance of speech recognition. In this paper, we propose a transfer learning mechanism to conduct topicaware recognition for dialogue speech. We first propose a new probabilistic topic model named Dialogue Speech Topic Model (DSTM) that is specialized for modeling the context of dialogue speech. We further propose a novel transfer learning mechanism for DSTM to significantly reduce its training cost while preserving its effectiveness for accurate topic inference. The experiment results demonstrate that proposed techniques in language model adaptation effectively improve the performance of the state-of-the-art Automatic Speech Recognition (ASR) system.

show abstract

Neural Network Language Modeling with Letter-Based Features and Importance Sampling

Cited by 54 publications

References 8 publications

Hit or Miss? Evaluating the Potential of a Research Niche: A Case Study in the Field of Virtual Quality Management

Hit or Miss? Evaluating the Potential of a Research Niche: A Case Study in the Field of Virtual Quality Management

The STC ASR System for the VOiCES from a Distance Challenge 2019

Topic-Aware Dialogue Speech Recognition with Transfer Learning

Contact Info

Product

Resources

About