Transformer-Based Acoustic Modeling for Streaming Speech Synthesis

Wu, Chunyang; Xiu, Zhiping; Shi, Yangyang; Kalinli, Ozlem; Fuegen, Christian; Koehler, Thilo; He, Qianhua

doi:10.21437/interspeech.2021-1655

Cited by 6 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A temporal network is modeled using the transformer architecture introduced in [27] to process the multi-feature fusion vectors generated in the previous step. Due to its capacity to capture global contextual information and parallelization capabilities [28][29][30][31], the transformer model is widely employed in the natural language processing and computer vision domains. Therefore, we employ the transformer model as one of the modules for extracting temporal information from the network.…”

Section: Feature Extractionmentioning

confidence: 99%

ARCnet: A Multi-Feature-Based Auto Radio Check Model

Pan,

Wang,

Zhang

et al. 2024

Aerospace

View full text Add to dashboard Cite

Radio checks serve as the foundation for ground-to-air communication. To integrate machine learning for automated and reliable radio checks, this study introduces an Auto Radio Check network (ARCnet), a novel algorithm for non-intrusive speech quality assessment in civil aviation, addressing the crucial need for dependable ground-to-air communication. By employing a multi-scale feature fusion approach, including the consideration of audio’s frequency domain, comprehensibility, and temporal information within the radio check scoring network, ARCnet integrates manually designed features with self-supervised features and utilizes a transformer network to enhance speech segment analysis. Utilizing the NISQA open-source dataset and the proprietary RadioCheckSpeech dataset, ARCnet demonstrates superior performance in predicting speech quality, showing a 12% improvement in both the Pearson correlation coefficient and root mean square error (RMSE) compared to existing models. This research not only highlights the significance of applying multi-scale attributes and deep neural network parameters in speech quality assessment but also emphasizes the crucial role of the temporal network in capturing the nuances of voice data. Through a comprehensive comparison of the ARCnet approach to traditional methods, this study underscores its innovative contribution to enhancing communication efficiency and safety in civil aviation.

show abstract

Section: Feature Extractionmentioning

confidence: 99%

ARCnet: A Multi-Feature-Based Auto Radio Check Model

Pan,

Wang,

Zhang

et al. 2024

Aerospace

View full text Add to dashboard Cite

show abstract

“…The frontend takes plain text as input and emits its phonetic representation (grapheme to phoneme) and an additional prosodic information for each token in the output sequence. These are fed into the acoustic backend that consists of a transformer-based prosody model which predicts phone-level F0 values and duration, and a transformer-based spectral model that produces frame-level mel-cepstral coefficients, F0 and periodicity features, and finally a sparse WaveRNN-based neural vocoder that predicts the final waveform [20]. The acoustic model is trained on a multi-speaker dataset consisting of 170 hours of TTS-quality audio across 96 English speakers, while the WaveRNN vocoder was trained for each speaker separately.…”

Section: Ttsmentioning

confidence: 99%

G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR

Wang

Cubuk

Rosenberg

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Aiming at reducing the reliance on expensive human annotations, data synthesis for Automatic Speech Recognition (ASR) has remained an active area of research. While prior work mainly focuses on synthetic speech generation for ASR data augmentation, its combination with text generation methods is considerably less explored. In this work, we explore text augmentation for ASR using large-scale pre-trained neural networks, and systematically compare those to traditional text augmentation methods. The generated synthetic texts are then converted to synthetic speech using a text-to-speech (TTS) system and added to the ASR training data. In experiments conducted on three datasets, we find that neural models achieve 9%-15% relative WER improvement and outperform traditional methods. We conclude that text augmentation, particularly through modern neural approaches, is a viable tool for improving the accuracy of ASR systems.

show abstract

“…Подобные нейронные сети нашли применение во всех основных задачах глубокого обучения -от обработки естественных языков [15] до автоматической сегментации изображений [16,17], постепенно вытесняя предыдущие поколения глубоких нейронных сетей, основанные на преобразованиях свертки и рекуррентных слоях. В задачах обработки речи преобразование самовнимания применялось для автоматического распознавания голоса [18], оценки эмоциональной окраски [19] и синтеза речи [20]. В задаче улучшения качества речи данный механизм использовался как замена части рекуррентных и сверточных преобразований [9].…”

Section: постановка задачиunclassified

Speech Enhancement Method Based on Modified Encoder-Decoder Pyramid Transformer

Lependin¹,

Nasretdinov²,

Ilyashenko³

2022

Proceedings of ISP RAS

View full text Add to dashboard Cite

The development of new technologies for voice communication has led to the need of improvement of speech enhancement methods. Modern users of information systems place high demands on both the intelligibility of the voice signal and its perceptual quality. In this work we propose a new approach to solving the problem of speech enhancement. For this, a modified pyramidal transformer neural network with an encoder-decoder structure was developed. The encoder compressed the spectrum of the voice signal into a pyramidal series of internal embeddings. The decoder with self-attention transformations reconstructed the mask of the complex ratio of the cleaned and noisy signals based on the embeddings calculated by the encoder. Two possible loss functions were considered for training the proposed neural network model. It was shown that the use of frequency encoding mixed with the input data has improved the performance of the proposed approach. The neural network was trained and tested on the DNS Challenge 2021 dataset. It showed high performance compared to modern speech enhancement methods. We provide a qualitative analysis of the training process of the implemented neural network. It showed that the network gradually moved from simple noise masking in the early training epochs to restoring the missing formant components of the speaker's voice in later epochs. This led to high performance metrics and subjective quality of enhanced speech.

show abstract

Transformer-Based Acoustic Modeling for Streaming Speech Synthesis

Cited by 6 publications

References 12 publications

ARCnet: A Multi-Feature-Based Auto Radio Check Model

ARCnet: A Multi-Feature-Based Auto Radio Check Model

G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR

Speech Enhancement Method Based on Modified Encoder-Decoder Pyramid Transformer

Contact Info

Product

Resources

About