Phonemic-level Duration Control Using Attention Alignment for Natural Speech Synthesis

Park, Jung-Bae; Han, Kijong; Jeong, Yuneui; Lee, Sang Wan

doi:10.1109/icassp.2019.8683827

Cited by 15 publications

(10 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of providing the Mel spectrogram of the reference audio as input to the reference encoder or variational framework, as is the case for all the systems mentioned above, specific prosodic features extracted from the reference audio, such as F0, duration and loudness, can be used as input to prosody embedding networks. These prosodic features and their statistics can be extracted at utterancelevel [11,12,13] or at frame-level and phoneme-level [14,15] to achieve utterance-level or fine-grained prosody control, respectively. A semi-supervised approach utilizing both Mel spectrograms and prosodic features as inputs to a variational framework is proposed in [16].…”

Section: Related Workmentioning

confidence: 99%

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Vioni¹,

Christidou²,

Ellinas³

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

This paper presents a method for controlling the prosody at the phoneme level in an autoregressive attention-based text-to-speech system. Instead of learning latent prosodic features with a variational framework as is commonly done, we directly extract phoneme-level F0 and duration features from the speech data in the training set. Each prosodic feature is discretized using unsupervised clustering in order to produce a sequence of prosodic labels for each utterance. This sequence is used in parallel to the phoneme sequence in order to condition the decoder with the utilization of a prosodic encoder and a corresponding attention module. Experimental results show that the proposed method retains the high quality of generated speech, while allowing phoneme-level control of F0 and duration. By replacing the F0 cluster centroids with musical notes, the model can also provide control over the note and octave within the range of the speaker.

show abstract

Section: Related Workmentioning

confidence: 99%

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Vioni¹,

Christidou²,

Ellinas³

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Explicit control over prosodic features such as F0 and duration by extracting these features and using them as input to a prosody encoder has been implemented for utterance-level [23,11,22] and more fine-grained [31,20,15,35] control.…”

Section: Related Workmentioning

confidence: 99%

Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control

Christidou,

Vioni,

Ellinas

et al. 2021

Preprint

View full text Add to dashboard Cite

This paper presents a method for phoneme-level prosody control of F0 and duration on a multispeaker text-to-speech setup, which is based on prosodic clustering. An autoregressive attention-based model is used, incorporating multispeaker architecture modules in parallel to a prosody encoder. Several improvements over the basic single-speaker method are proposed that increase the prosodic control range and coverage. More specifically we employ data augmentation, F0 normalization, balanced clustering for duration, and speaker-independent prosodic clustering. These modifications enable fine-grained phoneme-level prosody control for all speakers contained in the training set, while maintaining the speaker identity. The model is also fine-tuned to unseen speakers with limited amounts of data and it is shown to maintain its prosody control capabilities, verifying that the speaker-independent prosodic clustering is effective. Experimental results verify that the model maintains high output speech quality and that the proposed method allows efficient prosody control within each speaker's range despite the variability that a multispeaker setting introduces.

show abstract

“…Speech varies in expressions; however, these models only focus on the generation of narrative-style speech. Therefore, many researches have been recently proposed to control the prosody and speaking speed of the synthesized speech in a TTS system [5][6][7][8][9][10]. This paper focuses on the control of speaking speed that is essential for real scenario because the speaking speed must vary depending on the context or situation.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…In [9,10], neural TTS systems that control the phonemelevel speech duration have been proposed. Phoneme duration is additionally inputted to the TTS system [9], or the hidden states of the phoneme sequence are expanded, corresponding to the phoneme duration [10]. These systems, in the inference step, control the speaking speed by modifying the phoneme duration predicted by a duration predictor.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning

Bae¹,

Bae²,

Joo

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper proposes a controllable end-to-end text-to-speech (TTS) system to control the speaking speed (speed-controllable TTS; SCTTS) of synthesized speech with sentence-level speaking-rate value as an additional input. The speaking-rate value, the ratio of the number of input phonemes to the length of input speech, is adopted in the proposed system to control the speaking speed. Furthermore, the proposed SCTTS system can control the speaking speed while retaining other speech attributes, such as the pitch, by adopting the global style tokenbased style encoder. The proposed SCTTS does not require any additional well-trained model or an external speech database to extract phoneme-level duration information and can be trained in an end-to-end manner. In addition, our listening tests on fast-, normal-, and slow-speed speech showed that the SCTTS can generate more natural speech than other phoneme duration control approaches which increase or decrease duration at the same rate for the entire sentence, especially in the case of slow-speed speech.

show abstract

Phonemic-level Duration Control Using Attention Alignment for Natural Speech Synthesis

Cited by 15 publications

References 10 publications

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Prosodic Clustering for Phoneme-Level Prosody Control in End-to-End Speech Synthesis

Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Control

Speaking Speed Control of End-to-End Speech Synthesis using Sentence-Level Conditioning

Contact Info

Product

Resources

About