Text-Free Prosody-Aware Generative Spoken Language Modeling

Abstract

Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) (Lakhotia et al., 2021) is the only prior work addressing the generative aspects of speech pretraining, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic information. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. We devise a series of metrics for prosody modeling and generation, and re-use metrics from GSLM for content modeling. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. 1

GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models

“…In GSLM, a large autoregressive language model is typically trained on discovered discrete units (e.g. HuBERT [18] clusters or clustered spectrogram features), similar to how a language model is trained on text [19], [20]. While this also enables the generation of speech without any conditioning input, GSLM implies a model structure consisting of an encoder to discretize speech, a language model, and a decoder to convert the discrete units back into a waveform [17].…”

“…it is not possible to interpolate between two utterances in a latent space or to directly control speaker characteristics during generation. If this is desired, additional components must be explicitly built into the model [20].…”

Self-Supervised Language Learning From Raw Audio: Lessons From the Zero Resource Speech Challenge

“…The text is synthesized into speech. A spoken language model can be made to generate spoken language directly, as demonstrated by [67], [100], [126], [127]. Much as Task 3 is complementary to Task 1-but has slightly different constraints-the task of generating speech from a spoken language model is complementary to Task 4, yielding a potential Task 5.…”

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

“…Thus, they involve another phase of converting from the spectral domain to the time domain using a vocoder. Moreover, using discrete self-supervised speech representations and generating waveforms from these was demonstrated to provide superior performance on plenty of downstream tasks such as speech and audio language modelling Borsos et al, 2022;Qian et al, 2022), multi-stream processing (Kharitonov et al, 2022b), speech emotion conversion (Kreuk et al, 2021), spoken dialogue (Nguyen et al, 2022), speech-tospeech translation (Lee et al, 2022a,b;Popuri et al, 2022), and audio generation (Kreuk et al, 2022a,b).…”