A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion

Abstract

We propose a new paradigm for maintaining speaker identity in dysarthric voice conversion (DVC). The poor quality of dysarthric speech can be greatly improved by statistical VC, but as the normal speech utterances of a dysarthria patient are nearly impossible to collect, previous work failed to recover the individuality of the patient. In light of this, we suggest a novel, two-stage approach for DVC, which is highly flexible in that no normal speech of the patient is required. First, a powerful parallel sequence-to-sequence model converts the input dysarthric speech into a normal speech of a reference speaker as an intermediate product, and a nonparallel, frame-wise VC model realized with a variational autoencoder then converts the speaker identity of the reference speech back to that of the patient while assumed to be capable of preserving the enhanced quality. We investigate several design options. Experimental evaluation results demonstrate the potential of our approach to improving the quality of the dysarthric speech while maintaining the speaker identity.

Towards Identity Preserving Normal to Dysarthric Voice Conversion

“…The goal in the first stage is to completely capture the characteristics of the dysarthric speech. Following [6], we adopted the VTN [1,7], a Transformer-based [8] seq2seq model tailored for VC. When a parallel corpus is available, seq2seq modeling is considered state-ofthe-art due to its ability to convert the prosodic structures in speech, which is critical in N2D VC.…”

“…This technique is flexible in that the VC corpus and the pretraining TTS dataset can be completely different in terms of speaker and content, even when trained between normal and dysarthric speakers. In [6], it was shown that training using only 15 minutes of speech from each speaker can yield good results.…”

“…In the second stage, given the converted dysarthric speech, the goal is to restore the identity of the source normal speaker while preserving the dysarthric attributes. We adopted the same assumption as in [6]: a nonparallel frame-wise VC model changes only timeinvariant characteristics such as the speaker identity, while preserving time-variant characteristics, such as the pronunciation. As in [6], we used crank [9], an open-source VC software that combines recent advances in VQVAE [16]-based VC methods, including the use of hierarchical architectures, cyclic loss and adversarial training, to carry out the conversion of the speaker identity step.…”

“…The aim of this work is to evaluate the model in a more practical setting than [4] by taking normal speech as input, which alleviates the need of maintaining a pathological voice bank described there. Inspired by [6], the proposed method is a two-stage approach, as depicted in Figure 1b. In the first stage, to capture the unique temporal structure of dysarthric speech, we adopt the Voice Transformer Network (VTN) [1,7], a sequence-to-sequence (seq2seq) VC model based on the Transformer [8] architecture.…”

The Effectiveness of Time Stretching for Enhancing Dysarthric Speech for Improved Dysarthric Speech Recognition

“…Previous studies of dysarthric VC (DVC) have largely consisted of partial least squares regression-(PLS) [5], Gaussian mixture model (GMM) [4], or deep neural network-based (DNN) [6,7] parallel methods. There are also some methods that incorporate non-parallel VC methods as part of a parallel VC system [8,9].…”