Hider-Finder-Combiner: An Adversarial Architecture for General Speech Signal Modification

Abstract

We introduce a prototype system for modifying an arbitrary parameter of a speech signal. Unlike signal processing approaches that require dedicated methods for different parameters, our system can -in principle -modify any control parameter that the signal can be annotated with. Our system comprises three neural networks. The 'hider' removes all information related to the control parameter, outputting a hidden embedding. The 'finder' is an adversary used to train the 'hider', attempting to detect the value of the control parameter from the hidden embedding. The 'combiner' network recombines the hidden embedding with a desired new value of the control parameter. The input and output to the system are mel-spectrograms and we employ a neural vocoder to generate the output speech waveform. As a proof of concept, we use F0 as the control parameter. The system was evaluated in terms of control parameter accuracy and naturalness against a high quality signal processing method of F0 modification that also works in the spectrogram domain. We also show that, with modifications only to training data, the system is capable of modifying the 1 st and 2 nd vocal tract formants, showing progress towards universal signal modification.

Learning and controlling the source-filter representation of speech with a variational autoencoder

“…For deep latent representation learning methods, the challenge is to relate the learned representation to interpretable speech attributes. In Qian et al (2020) and Webber et al (2020), this interpretability is enforced by the design of the model. Qian et al (2020) proposed to use three independent encoder networks to decompose a speech signal into f 0 , timbre and rhythm latent representations.…”

“…Qian et al (2020) proposed to use three independent encoder networks to decompose a speech signal into f 0 , timbre and rhythm latent representations. Webber et al (2020) focused on controlling source-filter parameters in speech signals, where the ability to control a given parameter (e.g., f 0 ) is enforced explicitly using labeled data and adversarial learning. In this approach, each parameter to be controlled requires a dedicated training of the model.…”

Harmonic-Net: Fundamental Frequency and Speech Rate Controllable Fast Neural Vocoder

“…In the case of neural vocoders, the synthesis quality deteriorates when the input f o is not included in the range of the training data. Several approaches have been proposed to solve this problem [33], [34], [35], [36], [37], [38], [39]. In contrast to AR models [33], [34], [35], non-AR models [36], [37], [38], [39] can realize real-time inference.…”

“…Several approaches have been proposed to solve this problem [33], [34], [35], [36], [37], [38], [39]. In contrast to AR models [33], [34], [35], non-AR models [36], [37], [38], [39] can realize real-time inference. The neural source filter [36] introduces nonlinear filtering and dilated convolutional layers for parametrically generated source excitation signals corresponding to f o by source-filter modeling [40].…”