Discrete/continuous modelling of speaking style in HMM-based speech synthesis: design and evaluation

Obin, Nicolas; Lanchantin, Pierre; Lacheret, Anne; Rodet, Xavier

doi:10.21437/interspeech.2011-697

Cited by 11 publications

(1 citation statement)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Human behavior style is a socially meaningful clustering of features found within and across multiple modalities, specifically in linguistics (Campbell-Kibler et al, 2006 ), spoken behavior such as the speaking style conveyed by speech prosody (Obin, 2011 ; Moon et al, 2022 ), and nonverbal behavior such as hand gestures and body posture (Wagner et al, 2014 ; Obermeier et al, 2015 ).…”

Section: Introductionmentioning

confidence: 99%

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Fares¹,

Pélachaud²,

Obin³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two input modalities–Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) a speaker style encoder network that learns to generate a fixed-dimensional speaker embedding style from a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) a sequence-to-sequence synthesis network that synthesizes gestures based on the content of the input modalities—text and mel-spectrogram—of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.

show abstract

Section: Introductionmentioning

confidence: 99%

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Fares¹,

Pélachaud²,

Obin³

2023

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

Surgery of Speech Synthesis Models to Overcome the Scarcity of Training Data

Pierard

Erro

Hernáez

et al. 2016

Advances in Speech and Language Technologies for Iberian Languages

View full text Add to dashboard Cite

Exploiting Alternatives for Text-To-Speech Synthesis: From Machine to Human

Obin

Veaux

Lanchantin

2015

Speech Prosody in Speech Synthesis: Modeling and Generation of Prosody for High Quality and Flexible Speech Synthesis

View full text Add to dashboard Cite

The absence of alternatives/variants is a dramatical limitation of text-tospeech synthesis compared to the variety of human speech. This paper introduces the use of speech alternatives/variants in order to improve text-to-speech synthesis systems. Speech alternatives denote the variety of possibilities that a speaker has to pronounce a sentence -depending on linguistic constraints, specific strategies of the speaker, speaking style, and pragmatic constraints. During the training, symbolic and acoustic characteristics of a unit-selection speech synthesis system are statistically modelled with context-dependent parametric models (GMMs/HMMs). During the synthesis, symbolic and acoustic alternatives are exploited using a GENERAL-IZED VITERBI ALGORITHM (GVA) to determine the sequence of speech units used for the synthesis. Objective and subjective evaluations support evidence that the use of speech alternatives significantly improves speech synthesis over conventional speech synthesis systems. Beyond, speech alternatives can also be used to vary the speech synthesis for a given text. The proposed method can easily be extended to HMM-based speech synthesis. IntroductionToday, speech synthesis systems (unit-selection [1], HMM-based [2]) are able to produce natural synthetic speech from text. Over the last decade, research has mainly focus on the modelling of speech prosody -"the music of speech" (accent/phrasing, intonation/rhythm) -for text-to-speech synthesis. Among them, GMM/HMM (Gaussian Mixture Models, and Hidden Markov Models) are today the most popular methods used to model speech prosody. In particular, the modelling of speech prosody has gradually and durably moved from short-time representations ("frame-by-frame": [3,4,5,6,7]) to the use of larger-time representations [8,9,10,11]). Also, recent researches tend to introduce deep architecture systems to model more efficiently the complexity of speech (Deep Neural Networks [12]). However, current speech synthesis systems still suffer from a number of limitations 1 2 Nicolas Obin, Christophe Veaux, Pierre Lanchantin which consequence into the fact that the synthetic speech does not totally sound as "human". In particular, the absence of alternatives/variants in the synthetic speech is a dramatical limitation compared to the variety of human speech (see figure 14.1 for illustration): for a given text, the speech synthesis system will always produce exactly the same synthetic speech. A human speaker can use a variety of alternatives/variants to pronounce a text. This variety may induce variations in the symbolic (prosodic event: accent, phrasing) and acoustic (prosody: prosodic contour; segmental: articulation, co-articulation) speech characteristics. These alternatives depend on linguistic constraints, specific strategies of the speaker, speaking style, and pragmatic constraints. Current speech synthesis systems do not exploit this variety during statistical modelling or synthesis. During the training, the symbolic and acoustic speech characteristics are ...

show abstract

Discrete/continuous modelling of speaking style in HMM-based speech synthesis: design and evaluation

Cited by 11 publications

References 11 publications

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding

Surgery of Speech Synthesis Models to Overcome the Scarcity of Training Data

Exploiting Alternatives for Text-To-Speech Synthesis: From Machine to Human

Contact Info

Product

Resources

About