Novel Inception-GAN for Whispered-to-Normal Speech Conversion

Abstract

Recently, Convolutional Neural Networks (CNN)-based Generative Adversarial Networks (GANs) are used for Whisper-to-Normal Speech (i.e., WHSP2SPCH) conversion task. These CNN-based GANs are significantly difficult to train in terms of computational complexity. Goal of the generator in GAN is to map the features of the whispered speech to that of the normal speech efficiently. To improve the performance, we need to either tune the cost functions by changing hyperparameters associated with it or to make the generator more complex by adding more layers to the model. However, more complex architectures are prone to overfitting. Both solutions are time-consuming and computationally expensive. Hence, in this paper, we propose Inception-based GAN architecture (i.e., Inception-GAN). Our proposed architecture is quite stable and computationally less expensive while training. The proposed Inception-GAN outperforms existing CNN-based GAN architectures (CNN-GAN). Objective and subjective results are carried out using the proposed architectures on statistically meaningful whispered TIMIT (wTIMIT) corpus. For a speakerspecific evaluations, Inception-GAN shows 8.9% and 6.2% better perfomance objectively compared to the CNN-based GAN for male and female speaker, respectively.

Machine Learning Approaches for Whisper to Normal Speech Conversion

“…One of these, namely Lian et al (2020), was excluded from the survey due to the lack of availability of a full text version. Additionally, following the same criteria, 3 other papers were collected through Web of Science and/or Google Scholar, namely Niranjan et al (2020), Gao et al (2021), andPatel et al (2019). It is also worth noting that, although beyond the scope of this survey, the prior research in whisper-to-normal speech conversion is largely covered by the background sections within the selected papers.…”

“…In Patel et al (2019), the authors adapted the Inception modules from Szegedy et al (2015), proposing an Inception-GAN architecture, aimed at reducing computational complexity. In contrast with a typical CNN based GAN, also implemented as baseline for comparison purposes, the generator and discriminator parts of the GAN are made of stacks of 4 and 3 inception modules, respectively, and only 1 convolutional layer each.…”

“…Some version of a dataset derived from the TIMIT corpus was used in all cases included here. Specifically, whispered TIMIT (wTimit) 1 was used by Niranjan et al (2020), Patel et al (2021), Parmar et al (2019), andPatel et al (2019), while CSTR-NAM-TIMIT Plus was used by Gao et al (2021), Lian et al (2019a), Malaviya et al (2020), Pang et al (2020), Yu et al (2019), and Lian et al (2019b). The wTIMIT dataset uses the prompts in TIMIT, a well-known corpus often used for benchmarking in speech recognition, including 450 phonetically balanced sentences both in normal and whispered speech.…”

“…where F0 C and F0 R are the fundamental frequencies of each of the K time-aligned frames of the converted and the reference normal speech, respectively. This metric was adopted in Gao et al (2021) and Lian et al (2019a) and in all the AHOCODER based works, namely Parmar et al (2019), Malaviya et al (2020), Patel et al (2021), andPatel et al (2019), in which cases the RMSE was calculated over the log(F0) instead. Additionally, in Patel et al (2021), the Kullback-Leibler Divergence (KLD) and Jensen-Shannon Divergence (JSD) between predicted and original F0 for speaker-specific tasks were also considered.…”

CinC-GAN for Effective F0 prediction for Whisper-to-Normal Speech Conversion

“…Moreover, parallel data requires timealignment as pre-processing. In addition, traditional method uses 2-step sequential method for WHSP2SPCH conversion [25], [26]. For CycleGAN based conversion, in first step, one CycleGAN is trained for cepstral feature mapping of whisper to normal speech, and in second step, another CycleGAN is trained for F 0 prediction, which heavily relies on previously trained CycleGAN [25].…”