Expressive Voice Conversion: A Joint Framework for Speaker Identity and Emotional Style Transfer

Abstract

Traditional voice conversion (VC) has been focused on speaker identity conversion for speech with a neutral expression. We note that emotional expression plays an essential role in daily communication, and the emotional style of speech can be speaker-dependent. In this paper, we study the technique to jointly convert the speaker identity and speakerdependent emotional style, that is called expressive voice conversion. We propose a StarGAN-based framework to learn a many-to-many mapping across different speakers, that takes into account speaker-dependent emotional style without the need for parallel data. To achieve this, we condition the generator on emotional style encoding derived from a pre-trained speech emotion recognition (SER) model. The experiments validate the effectiveness of our proposed framework in both objective and subjective evaluations. To our best knowledge, this is the first study on expressive voice conversion.

IQDUBBING: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion

“…Despite recent progress, modeling prosody from expressive speech [10] for style transfer with voice conversion framework is still a challenging task. Besides linguistic information, transferring the source prosody to the target is vital for many voice conversion tasks, including automatic dubbing 1 for movies in which conversations are emotional in nature.…”