Dfsmn-San with Persistent Memory Model for Automatic Speech Recognition

Zhao, You; Su, Dan; Chen, Jie; Weng, Chao; Yu, Dong

doi:10.1109/icassp40776.2020.9053568

Cited by 14 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The acoustic model is set up according to the paper [16], which contains both the Chinese syllable and the English phone as modeling units. We also apply the multi-graph decoding strategy [17], which contains code-switch and English 1 and the code-switch dataset as Table 2, and the English is trained by the transcript of librispeech.…”

Section: Setupmentioning

confidence: 99%

Code-Switch Speech Rescoring with Monolingual Data

Liu

Cao

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In the automatic speech recognition (ASR) system, how to solve the problem of code-switch speech recognition has been a concern. Code-switch speech recognition is challenging due to data scarcity as well as diverse syntactic structures across languages. In this paper, we focus on the code-switch speech recognition in mainland China, which is obviously different from the Hong Kong and Southeast Asia area in linguistic characteristics. We propose a novel approach that only uses monolingual data for code-switch second-pass speech recognition which is also named language model rescoring. The approach converts the code-switch sentence to a monolingual sentence by a word mapping and language model determination step, therefore the issue of data scarcity is unnecessary to be considered. The word pairs during the word mapping step are generated by a fine-designed generation process that incorporates machine translation, word alignment, etc. We show that the proposed approach achieves an over 7.23% relative WER reduction from the naive monolingual language model (MLM) rescoring in our test set.

show abstract

Section: Setupmentioning

confidence: 99%

Code-Switch Speech Rescoring with Monolingual Data

Liu

Cao

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…First is the AISHELL-2 [18] dataset which contains 1000 hours of speech data from 1991 speakers. Second is a 10K hours multi-domain dataset [10]. We also augment the AISEHLL-1 and AISEHLL-2 training data with 2-fold speed perturbation [19] in the experiments.…”

Section: Datasetsmentioning

confidence: 99%

“…Recently, transformer architecture which has achieved its success in the natural language process (NLP) tasks has also been widely used in ASR systems [8,9], demonstrating its superior performance compared with the state-of-the-art models. Our previous work also proposed a variant of model architecture which combined DFSMN with self-attention networks (SAN), and further applied the memory augmenting method on the self-attention layer [10]. In summary, the performance improvement of ASR systems owes much to the dedicated hand-designed model architectures.…”

Section: Introductionmentioning

confidence: 99%

Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

In this paper, we explore the neural architecture search (NAS) for automatic speech recognition (ASR) systems. We conduct the architecture search on the small proxy dataset, and then evaluate the network, constructed from the searched architecture, on the large dataset. Specially, we propose a revised search space that theoretically facilitates the search algorithm to explore the architectures with low complexity. Extensive experiments show that: (i) the architecture learned in the revised search space can greatly reduce the computational overhead and GPU memory usage with mild performance degradation. (ii) the searched architecture can achieve more than 15% (average on the four test sets) relative improvements on the large dataset, compared with our best hand-designed DFSMN-SAN architecture. To the best of our knowledge, this is the first report of NAS results with a large scale dataset (up to 10K hours), indicating the promising application of NAS to industrial ASR systems.

show abstract

“…64 i-vectors were tested to be a good choice to provide diverse speaker information (Zhao et al, 2020), and applying on all encoder layers helps capture speaker knowledge from both low-level phonetic features and high-level global information. thermore, here we also compare our model with the first persistent memory model used in ASR (You et al, 2019), in which persistent memory vectors are randomly initialized and meant to capture general knowledge. Different from them, our model is to address the speaker mismatch issue.…”

Section: Adaptation For General Speakersmentioning

confidence: 99%

A Unified Speaker Adaptation Approach for ASR

Zhao¹,

Ni²,

Leung³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Transformer models have been used in automatic speech recognition (ASR) successfully and yields state-of-the-art results. However, its performance is still affected by speaker mismatch between training and test data. Further finetuning a trained model with target speaker data is the most natural approach for adaptation, but it takes a lot of compute and may cause catastrophic forgetting to the existing speakers. In this work, we propose a unified speaker adaptation approach consisting of feature adaptation and model adaptation. For feature adaptation, we employ a speakeraware persistent memory model which generalizes better to unseen test speakers by making use of speaker i-vectors to form a persistent memory. For model adaptation, we use a novel gradual pruning method to adapt to target speakers without changing the model architecture, which to the best of our knowledge, has never been explored in ASR. Specifically, we gradually prune less contributing parameters on model encoder to a certain sparsity level, and use the pruned parameters for adaptation, while freezing the unpruned parameters to keep the original model performance. We conduct experiments on the Librispeech dataset. Our proposed approach brings relative 2.74-6.52% word error rate (WER) reduction on general speaker adaptation. On target speaker adaptation, our method outperforms the baseline with up to 20.58% relative WER reduction, and surpasses the finetuning method by up to relative 2.54%. Besides, with extremely low-resource adaptation data (e.g., 1 utterance), our method could improve the WER by relative 6.53% with only a few epochs of training.

show abstract

Dfsmn-San with Persistent Memory Model for Automatic Speech Recognition

Cited by 14 publications

References 15 publications

Code-Switch Speech Rescoring with Monolingual Data

Code-Switch Speech Rescoring with Monolingual Data

Learned Transferable Architectures Can Surpass Hand-Designed Architectures for Large Scale Speech Recognition

A Unified Speaker Adaptation Approach for ASR

Contact Info

Product

Resources

About