Effective Emotion Transplantation in an End-to-End Text-to-Speech System

Abstract

In this paper, we propose an effective technique to transplant a source speaker's emotional expression to a new target speaker's voice within an end-to-end text-to-speech (TTS) framework. We modify an expressive TTS model pre-trained using a source speaker's emotional speech database to reflect the voice characteristics of a target speaker for which only a neutral speech database is available. We set two adaptation criteria to achieve this. One criterion is to minimize the reconstruction loss between the target speaker's recorded and synthesized speech, such that the synthesized speech has the target speaker's voice characteristics. The other criterion is to minimize the emotion loss between the emotion embedding vectors extracted from the reference expressive speech and the target speaker's synthesized expressive speech, which is essential to preserve expressiveness. Since the two criteria are applied alternately in the adaptation process, we are able to avoid the kind of bias issues frequently encountered in similar tasks. The proposed adaptation technique demonstrates more effective performance compared to conventional approaches in both quantitative and qualitative evaluations.

The VNPT-It Emotion Transplantation Approach for VLSP 2022

“…An adaptation method which can transfer expressive speech synthesis from one speaker to another [3] is used for this task. This can be highly effective when we have an emotional dataset from a source speaker and a neutral dataset from a target speaker.…”

“…The key obstacle of this approach is how to adjust the model so that it replicates the target speaker's voice while maintaining the capacity to express the desired emotions. Some adaptation approaches have been proposed recently, for example [3,9,11,12].…”