Transferring Neural Speech Waveform Synthesizers to Musical Instrument Sounds Generation

Zhao, Yi; Wang, Xin; Juvela, Lauri; Yamagishi, Junichi

doi:10.1109/icassp40776.2020.9053047

Cited by 11 publications

(4 citation statements)

References 14 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The DDSP autoencoder [1] falls into the first category as it generates control signals for a spectral modelling synthesiser [25]. The neural source-filter (NSF) approach [3,24,26] is in the second category. It learns a nonlinear filter that transforms a sinusoidal exciter to a target signal, guided by a control embedding generated by a separate encoder.…”

Section: Neural Audio Synthesismentioning

confidence: 99%

Neural Waveshaping Synthesis

Hayes¹,

Saitis²,

Fazekas³

2021

Preprint

View full text Add to dashboard Cite

We present the Neural Waveshaping Unit (NEWT): a novel, lightweight, fully causal approach to neural audio synthesis which operates directly in the waveform domain, with an accompanying optimisation (FastNEWT) for efficient CPU inference. The NEWT uses time-distributed multilayer perceptrons with periodic activations to implicitly learn nonlinear transfer functions that encode the characteristics of a target timbre. Once trained, a NEWT can produce complex timbral evolutions by simple affine transformations of its input and output signals. We paired the NEWT with a differentiable noise synthesiser and reverb and found it capable of generating realistic musical instrument performances with only 260k total model parameters, conditioned on F0 and loudness features. We compared our method to state-of-the-art benchmarks with a multistimulus listening test and the Fréchet Audio Distance and found it performed competitively across the tested timbral domains. Our method significantly outperformed the benchmarks in terms of generation speed, and achieved real-time performance on a consumer CPU, both with and without FastNEWT, suggesting it is a viable basis for future creative sound design tools.

show abstract

Section: Neural Audio Synthesismentioning

confidence: 99%

Neural Waveshaping Synthesis

Hayes¹,

Saitis²,

Fazekas³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition to the choice of data for the base model, the training strategy is also important and can make a difference to the quality of the output. For instance, [14] found that for copy synthesis of music, starting with vocoder models that had originally been trained on speech data and then finetuning using music data worked better even than training only on music data from scratch, indicating that useful additional information for music signals can be learned from speech. For multi-speaker text-to-speech synthesis, [7] found that warmstarting the multi-speaker model from a high-quality singlespeaker pretrained model can greatly reduce training time as compared to training a multi-speaker model from scratch, and that the warm-starting approach also provides the benefits of a larger vocabulary.…”

Section: Introductionmentioning

confidence: 99%

Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis

Cooper,

Wang,

Zhao

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We explore pretraining strategies including choice of base corpus with the aim of choosing the best strategy for zero-shot multi-speaker end-to-end synthesis. We also examine choice of neural vocoder for waveform synthesis, as well as acoustic configurations used for mel spectrograms and final audio output. We find that fine-tuning a multi-speaker model from found audiobook data that has passed a simple quality threshold can improve naturalness and similarity to unseen target speakers of synthetic speech. Additionally, we find that listeners can discern between a 16kHz and 24kHz sampling rate, and that WaveRNN produces output waveforms of a comparable quality to WaveNet, with a faster inference time.

show abstract

“…In our previous works on TTS, we used sine-based source signals because their periodicity can be accurately maintained in the generated voiced sounds [13]. In addition to the speech waveforms, the sine-based waveform also helps the NSF models to produce high-quality music signals for woodwind, string, and brass instruments [15].…”

Section: Introductionmentioning

confidence: 99%

Using Cyclic Noise as the Source Signal for Neural Source-Filter-based Speech Waveform Model

Wang

Yamagishi

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Neural source-filter (NSF) waveform models generate speech waveforms by morphing sine-based source signals through dilated convolution in the time domain. Although the sinebased source signals help the NSF models to produce voiced sounds with specified pitch, the sine shape may constrain the generated waveform when the target voiced sounds are less periodic. In this paper, we propose a more flexible source signal called cyclic noise, a quasi-periodic noise sequence given by the convolution of a pulse train and a static random noise with a trainable decaying rate that controls the signal shape. We further propose a masked spectral loss to guide the NSF models to produce periodic voiced sounds from the cyclic noise-based source signal. Results from a large-scale listening test demonstrated the effectiveness of the cyclic noise and the masked spectral loss on speaker-independent NSF models in copy-synthesis experiments on the CMU ARCTIC database.

show abstract

Transferring Neural Speech Waveform Synthesizers to Musical Instrument Sounds Generation

Cited by 11 publications

References 14 publications

Neural Waveshaping Synthesis

Neural Waveshaping Synthesis

Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi-Speaker End-to-End Speech Synthesis

Using Cyclic Noise as the Source Signal for Neural Source-Filter-based Speech Waveform Model

Contact Info

Product

Resources

About