Cross-Lingual Speaker Adaptation for Statistical Speech Synthesis Using Limited Data

Abstract

Cross-lingual speaker adaptation with limited adaptation data has many applications such as use in speech-to-speech translation systems. Here, we focus on cross-lingual adaptation for statistical speech synthesis (SSS) systems using limited adaptation data. To that end, we propose two techniques exploiting a bilingual Turkish-English speech database that we collected. In one approach, speaker-specific state-mapping is proposed for cross-lingual adaptation which performed significantly better than the baseline state-mapping algorithm in adapting the excitation parameter both in objective and subjective tests. In the second approach, eigenvoice adaptation is done in the input language which is then used to estimate the eigenvoice weights in the output language using weighted linear regression. The second approach performed significantly better than the baseline system in adapting the spectral envelope parameters both in objective and subjective tests.

Exploring the Role of Language Families for Building Indic Speech Synthesisers

“…There have been previous attempts to train generic voice models from different perspectives-polyglot synthesis [17]- [28], code-mixing 2 [23], [26], [29]- [36], cross-lingual voice conversion [37], [38], and data augmentation [21], [22], [25], [26], [28], [39]- [44]. Polyglot synthesis aims to synthesise texts of multiple languages in the voice of a single speaker.…”

A Systematic Review and Analysis of Multilingual Data Strategies in Text-to-Speech for Low-Resource Languages

“…These resulting values (n = 880) were used for analysis. [6], [7], [8], [9], [10], [11], [12] Hidden Markov Model synthesis (HMM) 7 [12], [13], [14], [15], [16], [17], [18] Neural network (non-S2S) synthesis (DNN) 9 [19], [20], [21], [22], [23], [24], [25], [26], [27] Sequence-to-sequence synthesis (S2S)…”

“…As such a measure did not yet exist, we created one. This measure, hereafter referred to as the MultiLingual Model Effect (MLME), was derived from the reported results as follows: [18], [21], [27] F0 RMSE (Root-Mean-Square Error) Acoustics [12], [18] V/UV (Voiced/UnVoiced error) Acoustics [12], [18] BAP (Band APeriodicities distortion) Acoustics [18] DTWMCD (Dynamic Time Warping Mel-Cepstral Distortion) Acoustics [19] L2 NSE (L2 Norm-Squared on log-Mel spectrogram) Acoustics [24] LF0 RMSE (Log F0 Root-Mean-Square Error) Acoustics [11] LSD (normalized Log Cepstral Distance) Acoustics [12] MGC RMSE (Mel-Generalized Cepstrum coefficients Root-Mean-Square Error) Acoustics [11] MSE (Mean-Square Error) Acoustics [13] CER (Character Error Rate) Intelligibility [21], [26] Intelligibility % (percentage of intelligible sentences) Intelligibility [7] SUS-Wacc (Word accuracy in Semantically Unpredictable Sentences)…”

“…Intelligibility [23] WER (Word Error Rate) Intelligibility [20] MOS (Mean Opinion Score) Naturalness/ Quality [6], [8], [9], [14], [15], [16], [17], [18], [21], [23] A/B Preference (preference rate b/w test & control) Quality [5], [10], [11], [12], [13]…”

“…Since all studies (except for [19] and [26]) mentioned this only in either time duration or number of utterances, an average estimation was needed to convert it to a common measurement. Following the descriptions provided in [10], [11], [12], [16], [17], [20], [21], [22], [23], [24], and [27], we obtained an average utterance length of the speech data sets used in these studies: 6.1 seconds. This was then used to convert all the training data quantities to the corresponding estimated number of utterances.…”