Structural KLD for Cross-Variety Speaker Adaptation in HMM-based Speech Synthesis

Toman, Markus; Pucher, Michael

doi:10.2316/p.2013.798-069

Cited by 5 publications

(3 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A well-known method for data augmentation is speaker adaptation, where the most common approach is to build an average voice model of multiple speakers and then adapt a model for new (target) speaker from it. Speaker adaptation is a wellresearched topic in HMM-based speech synthesis [4,5,6,7,8,9] but still relatively unexplored for DNN-based synthesis. Arik et al [10] found that speaker adaptation by fine-tuning (i.e.…”

Section: Data Augmentationmentioning

confidence: 99%

Data Requirements, Selection and Augmentation for DNN-based Speech Synthesis from Crowdsourced Data

2018

View full text Add to dashboard Cite

Crowdsourcing speech recordings provides unique opportunities and challenges for personalized speech synthesis as it allows gathering of large quantities of data but with a huge variety in quality. Manual methods for data selection and cleaning quickly become infeasible, especially when producing larger quantities of voices. We present and analyze approaches for data selection and augmentation to cope with this. For differently-sized training sets, we assess speaker adaptation by transfer learning, including layer freezing, and sentence selection using maximum likelihood of forced alignment. The methodological framework utilizes statistical parametric speech synthesis based on Deep Neural Networks (DNNs). We compare objective scores for 576 voice models, representing all condition combinations. For a constrained set of conditions we also present results from a subjective listening test. We show that speaker adaptation improves overall quality in nearly all cases, sentence selection helps detecting recording errors, and layer freezing proves to be ineffective in our system. We also found that while Mel-Cepstral Distortion (MCD) does not correlate with listener preference across the range of values, the most preferred voices also exhibited the lowest values for MCD. These findings have implications on scalable methods of customized voice building and clinical applications with sparse data.

show abstract

Section: Data Augmentationmentioning

confidence: 99%

Data Requirements, Selection and Augmentation for DNN-based Speech Synthesis from Crowdsourced Data

2018

View full text Add to dashboard Cite

show abstract

“…Typical applications of cross-lingual technologies include adapting speaker models between different dialects, accents, or variants of languages [12,13]. We believe that we can consider different expressiveness or speaking styles in the same fashion, and use our proposed transplantation technique combined with cross-language technologies with the purpose of transplanting paralinguistic features between languages.…”

Section: Cross-lingual Emotion Transplantationmentioning

confidence: 99%

Towards Cross-Lingual Emotion Transplantation

Lorenzo-Trueba

Barra-Chicote

Yamagishi

et al. 2014

Advances in Speech and Language Technologies for Iberian Languages

View full text Add to dashboard Cite

In this paper we introduce the idea of cross-lingual emotion transplantation. The aim is to lean the nuances of emotional speech in a source language for which we have enough data to adapt an acceptable quality emotional model by means of CSMAPLR adaptation, and then convert the adaptation function so it can be applied to a target language in a different target speaker while maintaining the speaker identity but adding emotional information. The conversion between languages is done at state level by measuring the KLD distance between the Gaussian distributions of all the states and linking the closest ones. Finally, as the cross-lingual transplantation of spectral emotions (mainly anger) was found out to introduce significant amounts of spectral noise, we show the results of applying three different techniques related to adaptation parameters that can be used to reduce the noise. The results are measured in an objective fashion by means of a bi-dimensional PCA projection of the KLD distances between the considered models (neutral models of both languages, reference emotion for both languages and transplanted emotional model for the target language).

show abstract

“…As soon as the dialect/sociolect of the user is detected, we can use interpolation to create a dialog system persona that fits the dialect/sociolect spoken by the user. In Toman et al (2013a), we presented a method for cross-variety speaker transformation based on HSMM state mapping (Wu et al, 2009). Transforming the voice of a speaker from one variety to another can be used as a basis for dialect interpolation.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised and phonologically controlled interpolation of Austrian German language varieties for speech synthesis

Toman

Pucher

Moosmüller

et al. 2015

Speech Communication

Self Cite

View full text Add to dashboard Cite

This paper presents an unsupervised method that allows for gradual interpolation between language varieties in statistical parametric speech synthesis using Hidden Semi-Markov Models (HSMMs). We apply dynamic time warping using Kullback-Leibler divergence on two sequences of HSMM states to find adequate interpolation partners. The method operates on state sequences with explicit durations and also on expanded state sequences where each state corresponds to one feature frame. In an intelligibility and dialect rating subjective evaluation of synthesized test sentences, we show that our method can generate intermediate varieties for three Austrian dialects (Viennese, Innervillgraten, Bad Goisern). We also provide an extensive phonetic analysis of the interpolated samples. The analysis includes input-switch rules, which cover historically different phonological developments of the dialects versus the standard language; and phonological processes, which are phonetically motivated, gradual, and common to all varieties. We present an extended method which linearly interpolates phonological processes but uses a step function for input-switch rules. Our evaluation shows that the integration of this kind of phonological knowledge improves dialect authenticity judgment of the synthesized speech, as performed by dialect speakers. Since gradual transitions between varieties are an existing phenomenon, we can use our methods to adapt speech output systems accordingly.

show abstract

Structural KLD for Cross-Variety Speaker Adaptation in HMM-based Speech Synthesis

Cited by 5 publications

References 15 publications

Data Requirements, Selection and Augmentation for DNN-based Speech Synthesis from Crowdsourced Data

Data Requirements, Selection and Augmentation for DNN-based Speech Synthesis from Crowdsourced Data

Towards Cross-Lingual Emotion Transplantation

Unsupervised and phonologically controlled interpolation of Austrian German language varieties for speech synthesis

Contact Info

Product

Resources

About