High quality voice conversion using prosodic and high-resolution spectral features

Abstract

Voice conversion methods have advanced rapidly over the last decade. Studies have shown that speaker characteristics are captured by spectral feature as well as various prosodic features. Most existing conversion methods focus on the spectral feature as it directly represents the timbre characteristics, while some conversion methods have focused only on the prosodic feature represented by the fundamental frequency. In this paper, a comprehensive framework using deep neural networks to convert both timbre and prosodic features is proposed. The timbre feature is represented by a high-resolution spectral feature. The prosodic features include F0, intensity and duration. It is well known that DNN is useful as a tool to model highdimensional features. In this work, we show that DNN initialized by our proposed autoencoder pretraining yields good quality DNN conversion models. This pretraining is tailor-made for voice conversion and leverages on autoencoder to capture the generic spectral shape of source speech. Additionally, our framework uses segmental DNN models to capture the evolution of the prosodic features over time. To reconstruct the converted speech, the spectral feature produced by the DNN model is combined with the three prosodic features produced by the DNN segmental models. Our experimental results show that the application of both prosodic and high-resolution spectral features leads to quality converted speech as measured by objective evaluation and subjective listening tests.

High-quality Voice Conversion Using Spectrogram-Based WaveNet Vocoder

“…The alignment is usually applied directly by Dynamic Time Wrapping (DTW) [17]. Also, there are techniques to get a more accurate feature alignment with the help of automatic speech recognition (ASR) techniques [18,14,19]. The aligned feature sequences x = x1, .., xT and y = y1, .., yT are then converted frame by frame in different methods (e.g.…”

Guest Editorial: Immersive Audio/Visual Systems

“…Voice conversion, modifying the recorded speech of a source speaker toward a given target speaker, is a popular way to achieve such voice personalization. In this special issue, Nguyen et al [8] provide a comprehensive voice conversion framework using deep neural networks to convert both timbre and prosodic features. Experiments show that the use of prosodic and high-resolution spectral features leads to high-quality converted speech.…”

Noise-Robust Voice Conversion Using High-Quefrency Boosting via Sub-Band Cepstrum Conversion and Fusion

“…VC may be categorized into two categories: one with parallel corpus (i.e., both source and target speakers utter the same sentences) and the other with non-parallel corpus (i.e., the target speakers utter different sentences from the source speaker) tasks according to whether the source and target speakers speak the same texts. Many existing approaches that have yielded conversion results with both high quality and high similarity are based on parallel data, such as Gaussian mixture models (GMM) [2,11], frequency warping (FW) [12][13][14], deep neural networks (DNN) [15][16][17], non-negative matrix factorization (NMF) [18,19], and so on. This paper also focuses on VC with parallel training data.…”