Phonetic posteriorgram-based voice conversion system to improve speech intelligibility of dysarthric patients

Zheng, Wei-Zhong; Han, Ji-Yan; Lee, Chen-Kai; Lin, Yu-Yi; Chang, Shu-Han; Lai, Ying‐Hui

doi:10.1016/j.cmpb.2021.106602

Cited by 18 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent methods based on the use of linguistic features PPGs and vocoders have also been proposed and have proven to be effective [10,[23][24][25][26]28]. PPGs are high-level contextual representations obtained from the posterior probabilities of each phonetic class using a speaker-independent ASR system.…”

Section: Related Workmentioning

confidence: 99%

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

Ezzine,

Di Martino,

Frikha

2023

Applied Sciences

View full text Add to dashboard Cite

We present an any-to-one voice conversion (VC) system, using an autoregressive model and LPCNet vocoder, aimed at enhancing the converted speech in terms of naturalness, intelligibility, and speaker similarity. As the name implies, non-parallel any-to-one voice conversion does not require paired source and target speeches and can be employed for arbitrary speech conversion tasks. Recent advancements in neural-based vocoders, such as WaveNet, have improved the efficiency of speech synthesis. However, in practice, we find that the trajectory of some generated waveforms is not consistently smooth, leading to occasional voice errors. To address this issue, we propose to use an autoregressive (AR) conversion model along with the high-fidelity LPCNet vocoder. This combination not only solves the problems of waveform fluidity but also produces more natural and clear speech, with the added capability of real-time speech generation. To precisely represent the linguistic content of a given utterance, we use speaker-independent PPG features (SI-PPG) computed from an automatic speech recognition (ASR) model trained on a multi-speaker corpus. Next, a conversion model maps the SI-PPG to the acoustic representations used as input features for the LPCNet. The proposed autoregressive structure enables our system to produce the following prediction step outputs from the acoustic features predicted in the previous step. We evaluate the effectiveness of our system by performing any-to-one conversion pairs between native English speakers. Experimental results show that the proposed method outperforms state-of-the-art systems, producing higher speech quality and greater speaker similarity.

show abstract

Section: Related Workmentioning

confidence: 99%

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

Ezzine,

Di Martino,

Frikha

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Fig. 1 depicts the proposed DVC 3.1 system, which was obtained by modifying the system proposed in our previous study [16]. DVC 3.1 involves four stages: data augmentation, speaker-dependent automatic speech recognition (SD-ASR) [25] training, conversion model training, and conversion.…”

Section: A Proposed Architecturementioning

confidence: 99%

“…However, there is still room for improvement in terms of the temporal variability and instability of patients' speech. We recently proposed a deep learning-based DVC system (DVC 3.0) [16]. The system addresses the variability of patients' speech characteristics using the speaker-independent property of phonetic posteriorgrams (PPGs) [17,18] and converts phonemes into normal speech using a gated convolutional neural network model (gated CNN) with long-term memory effects.…”

mentioning

confidence: 99%

Improving the Efficiency of Dysarthria Voice Conversion System Based on Data Augmentation

Zheng,

Han,

Chen

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

Self Cite

View full text Add to dashboard Cite

Dysarthria, a speech disorder often caused by neurological damage, compromises the control of vocal muscles in patients, making their speech unclear and communication troublesome. Recently, voice-driven methods have been proposed to improve the speech intelligibility of patients with dysarthria. However, most methods require a significant representation of both the patient's and target speaker's corpus, which is problematic. This study aims to propose a data augmentationbased voice conversion (VC) system to reduce the recording burden on the speaker. We propose dysarthria voice conversion 3.1 (DVC 3.1) based on a data augmentation approach, including text-to-speech and StarGAN-VC architecture, to synthesize a large target and patient-like corpus to lower the burden of recording. An objective evaluation metric of the Google automatic speech recognition (Google ASR) system and a listening test were used to demonstrate the speech intelligibility benefits of DVC 3.1 under free-talk conditions. The DVC system without data augmentation (DVC 3.0) was used for comparison. Subjective and objective evaluation based on the experimental results indicated that the proposed DVC 3.1 system enhanced the Google ASR of two dysarthria patients by approximately [62.4%, 43.3%] and [55.9%, 57.3%] compared to unprocessed dysarthria speech and the DVC 3.0 system, respectively. Further, the proposed DVC 3.1 increased the speech intelligibility of two dysarthria patients by approximately [54.2%, 22.3%] and [63.4%, 70.1%] compared to unprocessed dysarthria speech and the DVC 3.0 system, respectively. The proposed DVC 3.1 system offers significant potential to improve the speech intelligibility performance of patients with dysarthria and enhance verbal communication quality.

show abstract

“…Other voice conversion methods include text-based approaches. The best-known text-based approach uses automatic speech recognition (ASR) models to extract phonetic posteriograms (PPGs), which are then used as linguistic information [23,24]. Text-based approaches that use ASR models have accurate linguistic information, are unlikely to be corrupted during voice conversion, and can even perform voice conversion between speakers of different languages if the ASR model used supports multiple languages.…”

Section: Introductionmentioning

confidence: 99%

Perturbation AUTOVC: Voice Conversion From Perturbation and Autoencoder Loss

Park,

Lee,

Chun

2023

IEEE Access

View full text Add to dashboard Cite

AUTOVC is a voice-conversion method that performs self-reconstruction using an autoencoder structure for zero-shot voice conversion. AUTOVC has the advantage of being easy and simple to learn because it only uses the autoencoder loss for learning. However, it performs voice conversion by disentangling speech information from speakers and linguistic information by adjusting the bottleneck dimension; this requires highly meticulous fine tuning of the bottleneck dimension and involves a tradeoff between speech quality and speaker similarity. To address these issues, neural analysis and synthesis (NANSY)-a fully self-supervised learning system that uses perturbations to extract speech features-is proposed. NANSY solves the problem of the adjustment of the bottleneck dimension by utilizing perturbation and exhibits high-reconstruction performance. In this study, we propose perturbation AUTOVC, a voice conversion method that utilizes the structure of AUTOVC and the perturbation of NANSY. The proposed method applies perturbations to speech signals (such as NANSY signals) to solve the problem of the voice conversion method using bottleneck dimensions. Perturbation is applied to remove the speaker-dependent information present in the speech, leaving only the linguistic information, which is then passed through a content encoder and modeled as a content embedding containing only the linguistic information. To obtain speaker information, we used x-vectors, which are extensively used in pretrained speaker recognition. The concatenated linguistic and speaker information extracted from the encoder and additional energy information is used as input to the decoder to perform self-reconstruction. Similar to AUTOVC, it is easy and simple to learn using only the autoencoder loss. For the evaluation, we measured three objective evaluation metrics: character error rate (%), cosine similarity, and short-time objective intelligibility, as well as a subjective evaluation metric: mean opinion score. The experimental results demonstrate that our proposed method outperforms other voice conversion techniques and demonstrated robust performance in zero-shot conversion.

show abstract

Phonetic posteriorgram-based voice conversion system to improve speech intelligibility of dysarthric patients

Cited by 18 publications

References 10 publications

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

Any-to-One Non-Parallel Voice Conversion System Using an Autoregressive Conversion Model and LPCNet Vocoder

Improving the Efficiency of Dysarthria Voice Conversion System Based on Data Augmentation

Perturbation AUTOVC: Voice Conversion From Perturbation and Autoencoder Loss

Contact Info

Product

Resources

About