You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation

Laptev, Aleksandr; Korostik, Roman; Svischev, Aleksey; Andrusenko, Andrei; Medennikov, Ivan; Rybin, Sergey V.

doi:10.1109/cisp-bmei51763.2020.9263564

Cited by 50 publications

(38 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the quality of recent TTS systems has reached a natural level, several attempts have been made to apply TTS-synthesized speech databases to speech applications. For instance, Laptev et al [12] and Jia et al [13] improved the performance of automatic speech recognition and speech translation systems by training models with synthetic speech databases generated by Tacotron. In the TTS applications, Sharma et al [14] showed that the AR WaveNet-driven data augmentation is effective for improving the quality of Parallel WaveNet system [15].…”

Section: Relationship To Prior Workmentioning

confidence: 99%

TTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis

Hwang

Yamamoto

Song

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this paper, we propose a text-to-speech (TTS)-driven data augmentation method for improving the quality of a non-autoregressive (AR) TTS system. Recently proposed non-AR models, such as FastSpeech 2, have successfully achieved fast speech synthesis system. However, their quality is not satisfactory, especially when the amount of training data is insufficient. To address this problem, we propose an effective data augmentation method using a welldesigned AR TTS system. In this method, large-scale synthetic corpora including text-waveform pairs with phoneme duration are generated by the AR TTS system, and then used to train the target non-AR model. Perceptual listening test results showed that the proposed method significantly improved the quality of the non-AR TTS system. In particular, we augmented five hours of a training database to 179 hours of a synthetic one. Using these databases, our TTS system consisting of a FastSpeech 2 acoustic model with a Parallel WaveGAN vocoder achieved a mean opinion score of 3.74, which is 40% higher than that achieved by the conventional method.

show abstract

Section: Relationship To Prior Workmentioning

confidence: 99%

TTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis

Hwang

Yamamoto

Song

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Employing synthetic audio from TTS for ASR training recently gained popularity as a result of advancements in TTS. Recent research [11][12][13] has studied creating acoustically and lexically diverse synthetic data, exploring the feasibility of replacing or augmenting real recordings with synthetic data during ASR model training, without compromising recognition performance. The results show that synthetic audio can improve the training convergence when the amount of available real data is as small as 10 hours but does not yet replace real speech recordings to achieve the same recognition performance given the same text sources [11].…”

Section: Related Workmentioning

confidence: 99%

Using Synthetic Audio to Improve the Recognition of Out-of-Vocabulary Words in End-to-End Asr Systems

Zheng

Liu

Gunceler

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Today, many state-of-the-art automatic speech recognition (ASR) systems apply all-neural models that map audio to word sequences trained end-to-end along one global optimisation criterion in a fully data driven fashion. These models allow high precision ASR for domains and words represented in the training material but have difficulties recognising words that are rarely or not at all represented during training, i.e. trending words and new named entities. In this paper, we use a text-to-speech (TTS) engine to provide synthetic audio for out-of-vocabulary (OOV) words. We aim to boost the recognition accuracy of a recurrent neural network transducer (RNN-T) on OOV words by using the extra audio-text pairs, while maintaining the performance on the non-OOV words. Different regularisation techniques are explored and the best performance is achieved by fine-tuning the RNN-T on both original training data and extra synthetic data with elastic weight consolidation (EWC) applied on the encoder. This yields a 57% relative word error rate (WER) reduction on utterances containing OOV words without any degradation on the whole test set.

show abstract

“…A multi-head self-attention (MHA) mechanism significantly improved the quality of models over the recurrent models. A transformer model, trained with CTC-Attention, can outperform neural transducer systems (e.g., [ 36 ]) and benefit from various augmentation techniques [ 37 ]. Recently, the Conformer [ 38 ] was introduced, which is a modification of the transformer layer.…”

Section: Asr Modelingmentioning

confidence: 99%

Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition

Laptev

Andrusenko

Podluzhny

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

With the rapid development of speech assistants, adapting server-intended automatic speech recognition (ASR) solutions to a direct device has become crucial. For on-device speech recognition tasks, researchers and industry prefer end-to-end ASR systems as they can be made resource-efficient while maintaining a higher quality compared to hybrid systems. However, building end-to-end models requires a significant amount of speech data. Personalization, which is mainly handling out-of-vocabulary (OOV) words, is another challenging task associated with speech assistants. In this work, we consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate, embodied in Babel Turkish and Babel Georgian tasks. We propose a method of dynamic acoustic unit augmentation based on the Byte Pair Encoding with dropout (BPE-dropout) technique. The method non-deterministically tokenizes utterances to extend the token’s contexts and to regularize their distribution for the model’s recognition of unseen words. It also reduces the need for optimal subword vocabulary size search. The technique provides a steady improvement in regular and personalized (OOV-oriented) speech recognition tasks (at least 6% relative word error rate (WER) and 25% relative F-score) at no additional computational cost. Owing to the BPE-dropout use, our monolingual Turkish Conformer has achieved a competitive result with 22.2% character error rate (CER) and 38.9% WER, which is close to the best published multilingual system.

show abstract

You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation

Cited by 50 publications

References 29 publications

TTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis

TTS-by-TTS: TTS-Driven Data Augmentation for Fast and High-Quality Speech Synthesis

Using Synthetic Audio to Improve the Recognition of Out-of-Vocabulary Words in End-to-End Asr Systems

Dynamic Acoustic Unit Augmentation with BPE-Dropout for Low-Resource End-to-End Speech Recognition

Contact Info

Product

Resources

About