Comparing acoustic and textual representations of previous linguistic context for improving Text-to-Speech

Oplustil-Gallegos, Pilar; O'Mahony, Johannah; King, Simon

doi:10.21437/ssw.2021-36

Cited by 7 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Growing research interest in expressive speech synthesis for conversational speech has led towards efforts solving two subproblems. On the one hand, recent research has looked into building context-aware neural TTS models by incorporating and conditioning contextual information such as audio and linguistic features during training [6,7]. This has also led to newly proposed evaluation paradigms, which aim to move from rating naturalness of speech in isolation to rating appropriateness of speech in context [8].…”

Section: Related Workmentioning

confidence: 99%

Strategies for developing a Conversational Speech Dataset for Text-To-Speech Synthesis

Adigwe¹,

Klabbers²

2022

Interspeech 2022

View full text Add to dashboard Cite

There have been many efforts to improve the quality of speech synthesis systems in conversational AI. Although state-of-theart systems are capable of producing natural-sounding speech, the generated speech often lacks prosodic variation and is not always suited to the task. In this paper, we examine dialogue data collection methods to use as training data for our acoustic models. We collect speech using three different setups: (1) Random read-aloud sentences; (2) Performed dialogues; (3) Semi-Spontaneous dialogues. We analyze prosodic and textual properties of the data collected in these setups and make some recommendations to collect data for speech synthesis in conversational AI settings.

show abstract

Section: Related Workmentioning

confidence: 99%

Strategies for developing a Conversational Speech Dataset for Text-To-Speech Synthesis

Adigwe¹,

Klabbers²

2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…Context is a broad term and has different effects on an utterance e.g., pragmatic context [6], entrainment with a speaking partner or utterance position in a turn [7]. The effect of context on speech, especially its prosodic realisation, is poorly understood and can be data-dependent [8]. Prosodic cues also show inter-speaker variability [9].…”

Section: Introductionmentioning

confidence: 99%

Synthesising turn-taking cues using natural conversational data

O'Mahony,

Lai,

King

2023

12th ISCA Speech Synthesis Workshop (SSW2023)

View full text Add to dashboard Cite

As speech synthesis quality reaches high levels of naturalness for isolated utterances, more work is focusing on the synthesis of context-dependent conversational speech. The role of context in conversation is still poorly understood and many contextual factors can affect an utterances's prosodic realisation. Most studies incorporating context use rich acoustic or textual embeddings of the previous context, then demonstrate improvements in overall naturalness. Such studies are not informative about what the context embedding represents, or how it affects an utterance's realisation. So instead, we narrow the focus to a single, explicit contextual factor. In the current work, this is turn-taking. We condition a speech synthesis model on whether an utterance is turn-final. Objective measures and targeted subjective evaluation are used to demonstrate that the model can synthesise turn-taking cues which are perceived by listeners, with results being speaker-dependent.

show abstract

“…Apart from the textual context, it has been reported that acoustic features from the previous sentence can also lead to improvement of sentence-based TTS [43]. Following this result, a comparison investigation on multiple context representation types of the previous sentence was studied [44], including textual and acoustic features, utterance-level and word-level features, and representations extracted with a large pre-trained model and learned jointly with the TTS training. Our work differs from these in two folds: (i) the contextual information used in these models was derived from either isolation sentences or consecutive sentences with a predefined length, while in our work, the contextual information is extracted from a variable length paragraph which is a self-contained unit of discourse and composed of several connective sentences; (ii) crosssentence linguistic context were used to improve sentencelevel or conversation-level speech synthesis in their models.…”

Section: Introductionmentioning

confidence: 99%

ParaTTS: Learning Linguistic and Prosodic Cross-Sentence Information in Paragraph-Based TTS

Xue

Soong

Zhang

et al. 2022

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Recent advancements in neural end-to-end TTS models have shown high-quality, natural synthesized speech in a conventional sentence-based TTS. However, it is still challenging to reproduce similar high quality when a whole paragraph is considered in TTS, where a large amount of contextual information needs to be considered in building a paragraph-based TTS model. To alleviate the difficulty in training, we propose to model linguistic and prosodic information by considering cross-sentence, embedded structure in training. Three sub-modules, including linguistics-aware, prosody-aware and sentence-position networks, are trained together with a modified Tacotron2. Specifically, to learn the information embedded in a paragraph and the relations among the corresponding component sentences, we utilize linguistics-aware and prosody-aware networks. The information in a paragraph is captured by encoders and the inter-sentence information in a paragraph is learned with multi-head attention mechanisms. The relative sentence position in a paragraph is explicitly exploited by a sentence-position network. Trained on a storytelling audio-book corpus (4.08 hours), recorded by a female Mandarin Chinese speaker, the proposed TTS model demonstrates that it can produce rather natural and goodquality speech paragraph-wise. The cross-sentence contextual information, such as break and prosodic variations between consecutive sentences, can be better predicted and rendered than the sentence-based model. Tested on paragraph texts, of which the lengths are similar to, longer than, or much longer than the typical paragraph length of the training data, the TTS speech produced by the new model is consistently preferred over the sentence-based model in subjective tests and confirmed in objective measures.

show abstract

Comparing acoustic and textual representations of previous linguistic context for improving Text-to-Speech

Cited by 7 publications

References 0 publications

Strategies for developing a Conversational Speech Dataset for Text-To-Speech Synthesis

Strategies for developing a Conversational Speech Dataset for Text-To-Speech Synthesis

Synthesising turn-taking cues using natural conversational data

ParaTTS: Learning Linguistic and Prosodic Cross-Sentence Information in Paragraph-Based TTS

Contact Info

Product

Resources

About