Extracting and Predicting Word-Level Style Variations for Speech Synthesis

Zhang, Yajie; Ling, Zhen-Hua

doi:10.1109/taslp.2021.3074757

Cited by 11 publications

(5 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…WSV* Word-level style variations (WSV) model. For a fair comparison, instead of Tacotron2 [1] used in the original version of WSV [15], FastSpeech 2 was adopted as the backbone in our implementation. In addition, an extra bidirectional GRU is used to consider the context information.…”

Section: Compared Methodsmentioning

confidence: 99%

“…The root mean square error (RMSE) of F0 and energy, and the MSE of duration are adopted as the metrics of objective evaluation following [3,15]. To calculate the RMSE of F0 and energy, we first apply the dynamic time warping (DTW) to construct the alignment paths between the predicted mel-spectrogram and the ground-truth one.…”

Section: Objective Evaluationmentioning

confidence: 99%

“…However, HCE still suffers from the absence of local-scale style modeling (e.g., intonation, rhythm, stress). To model and control local prosodic variations in speech, some previous works attempt to predict finer-grained speaking styles from text, such as word level [15] and phoneme level [16]. It is more widely accepted that the style expressions of human speech are multi-scale in nature [17,18], where the global-scale style is usually observed as emotion and the local-scale is more close to the prosody variation [19,20].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Lei¹,

Zhou²,

Chen³

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Previous works on expressive speech synthesis focus on modelling the mono-scale style embedding from the current sentence or context, but the multi-scale nature of speaking style in human speech is neglected. In this paper, we propose a multiscale speaking style modelling method to capture and predict multi-scale speaking style for improving the naturalness and expressiveness of synthetic speech. A multi-scale extractor is proposed to extract speaking style embeddings at three different levels from the ground-truth speech, and explicitly guide the training of a multi-scale style predictor based on hierarchical context information. Both objective and subjective evaluations on a Mandarin audiobooks dataset demonstrate that our proposed method can significantly improve the naturalness and expressiveness of the synthesized speech 1 .

show abstract

Section: Compared Methodsmentioning

confidence: 99%

Section: Objective Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Lei¹,

Zhou²,

Chen³

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Luz's improved system could generate richer prosodic speech during the inference stage with limited training data [15]. Zhang and Ling proposed a speech synthesis model based on fine-grained style representation, called word-level style variation (WSV) [16]. In order to improve the accuracy of WSV prediction and the naturalness of synthesized speech, Zhang and Ling used a pretrained BERT model and speech information to derive semantic descriptions.…”

Section: Ttsmentioning

confidence: 99%

An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa

Yu,

Zhou,

Niu

2023

Applied Sciences

View full text Add to dashboard Cite

The prediction of pause fillers plays a crucial role in enhancing the naturalness of synthesized speech. In recent years, neural networks, including LSTM, BERT, and XLNet, have been employed for pause fillers prediction modules. However, these methods have exhibited relatively lower accuracy in predicting pause fillers. This paper introduces the utilization of the RoBERTa model for predicting Chinese pause fillers and presents a novel approach to training the RoBERTa model, effectively enhancing the accuracy of Chinese pause fillers prediction. Our proposed approach involves categorizing text from different speakers into four distinct style groups based on the frequency and position of Chinese pause fillers. The RoBERTa model is trained on these four groups of data, which incorporate different styles of fillers, thereby ensuring a more natural synthesis of speech. The Chinese pause fillers prediction module is evaluated on systems such as Parallel Tacotron2, FastPitch, and Deep Voice3, achieving a notable 26.7% improvement in word-level prediction accuracy compared to the BERT model, along with a 14% enhancement in position-level prediction accuracy. This substantial improvement results in a significant enhancement of the naturalness of the generated speech.

show abstract

“…Furthermore, to analyze the explainability of MFN, the dynamic fusion graph model (DFG) is embedded into MFN, and a Graph-MFN obtained finally has excellent performance and is explainable [ 10 ]. Recently, word-level fusion representation has also been a wide concern [ 23 ]. For example, a repeated participation variation network (RAVEN) is used to model multimodal language through work representation transfer based on facial expression [ 24 ].…”

Section: Correlation Workmentioning

confidence: 99%

Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis

Yanga

Zhang

et al. 2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Multimodal sentiment analysis (MSA) aims to infer emotions from linguistic, auditory, and visual sequences. Multimodal information representation method and fusion technology are keys to MSA. However, the problem of difficulty in fully obtaining heterogeneous data interactions in MSA usually exists. To solve these problems, a new framework, namely, dynamic invariant-specific representation fusion network (DISRFN), is put forward in this study. Firstly, in order to effectively utilize redundant information, the joint domain separation representations of all modes are obtained through the improved joint domain separation network. Then, the hierarchical graph fusion net (HGFN) is used for dynamically fusing each representation to obtain the interaction of multimodal data for guidance in the sentiment analysis. Moreover, comparative experiments are performed on popular MSA data sets MOSI and MOSEI, and the research on fusion strategy, loss function ablation, and similarity loss function analysis experiments is designed. The experimental results verify the effectiveness of the DISRFN framework and loss function.

show abstract

Extracting and Predicting Word-Level Style Variations for Speech Synthesis

Cited by 11 publications

References 35 publications

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa

Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis

Contact Info

Product

Resources

About