SoundStream: An End-to-End Neural Audio Codec

Zeghidour, Neil; Luebs, Alejandro; Omran, Ahmed; Skoglund, Jan; Tagliasacchi, Marco

doi:10.1109/taslp.2021.3129994

Cited by 209 publications

(133 citation statements)

References 40 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…In our experiments, we use a neural vocoder to perform the audio synthesis from log-mel spectrograms. The architecture of the vocoder is identical to MelGAN [12], but it is trained with a multi-scale reconstruction loss as well as adversarial losses from both wave-and STFT-based discriminators, following [22].…”

Section: Modelmentioning

confidence: 99%

“…At the same time, we optimize G to produce outputs that are indistinguishable from the ground truth by matching the feature representations in all layers of D [13,12]. D is a convolutional network closely resembling the single-scale STFT discriminator of [22], shown in Figure 2 -for further details about the architecture, we refer the reader to [22]. The discriminator is trained with the hinge loss,…”

Section: Two-phase Trainingmentioning

confidence: 99%

See 1 more Smart Citation

SpeechPainter: Text-conditioned Speech Inpainting

Borsos¹,

Sharifi²,

Tagliasacchi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We propose SpeechPainter, a model for filling in gaps of up to one second in speech samples by leveraging an auxiliary textual input. We demonstrate that the model performs speech inpainting with the appropriate content, while maintaining speaker identity, prosody and recording environment conditions, and generalizing to unseen speakers. Our approach significantly outperforms baselines constructed using adaptive TTS, as judged by human raters in side-by-side preference and MOS tests.

show abstract

Section: Modelmentioning

confidence: 99%

Section: Two-phase Trainingmentioning

confidence: 99%

SpeechPainter: Text-conditioned Speech Inpainting

Borsos¹,

Sharifi²,

Tagliasacchi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…(Agustsson et al, 2019;Mentzer et al, 2020). Neural compression has also been applied to video (Lu et al, 2019;Goliński et al, 2020;Agustsson et al, 2020) and audio (Kleijn et al, 2018;Valin & Skoglund, 2019;Yang et al, 2019;Zeghidour et al, 2021).…”

Section: Entropy Codingmentioning

confidence: 99%

COIN++: Neural Compression Across Modalities

Dupont¹,

Loya²,

Alizadeh³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural compression algorithms are typically based on autoencoders that require specialized encoder and decoder architectures for different data modalities. In this paper, we propose COIN++, a neural compression framework that seamlessly handles a wide range of data modalities. Our approach is based on converting data to implicit neural representations, i.e. neural functions that map coordinates (such as pixel locations) to features (such as RGB values). Then, instead of storing the weights of the implicit neural representation directly, we store modulations applied to a meta-learned base network as a compressed code for the data. We further quantize and entropy code these modulations, leading to large compression gains while reducing encoding time by two orders of magnitude compared to baselines. We empirically demonstrate the effectiveness of our method by compressing various data modalities, from images to medical and climate data.

show abstract

“…The model architecture is based on the decoder described in [27,28], which is a real-time streaming-capable version of MelGAN [29]. Its structure with parameters is shown in Table 1.…”

Section: Neural Network Based Spectrogram Inversionmentioning

confidence: 99%

“…We train the neural vocoder with the same mix of losses used in [28] to achieve both signal reconstruction fidelity and perceptual quality, following the principles of the perceptiondistortion trade-off discussed in [31]. The adversarial loss is used to promote perceptual quality and it is defined as a hinge loss over the logits of the discriminator, averaged over multiple discriminators and over time, operating both in the time domain and in the STFT domain.…”

Section: Neural Network Based Spectrogram Inversionmentioning

confidence: 99%

Real time spectrogram inversion on mobile phone

Rybakov¹,

Tagliasacchi²,

Liu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With the growth of computing power on mobile phones and privacy concerns over user's data, on-device real time speech processing has become an important research topic. In this paper, we focus on methods for real time spectrogram inversion, where an algorithm receives a portion of the input signal (e.g., one frame) and processes it incrementally, i.e., operating in streaming mode. We present a real time Griffin Lim(GL) algorithm using a sliding window approach in STFT domain. The proposed algorithm is 2.4x faster than real time on the ARM CPU of a Pixel4. In addition we explore a neural vocoder operating in streaming mode and demonstrate the impact of looking ahead on perceptual quality. As little as one hop size (12.5ms) of lookahead is able to significantly improve perceptual quality in comparison to a causal model. We compare GL with the neural vocoder and show different trade-offs in terms of perceptual quality, on-device latency, algorithmic delay, memory footprint and noise sensitivity. For fair quality assessment of the GL approach, we use input log magnitude spectrogram without mel transformation. We evaluate presented real time spectrogram inversion approaches on clean, noisy and atypical speech.

show abstract

SoundStream: An End-to-End Neural Audio Codec

Cited by 209 publications

References 40 publications

SpeechPainter: Text-conditioned Speech Inpainting

SpeechPainter: Text-conditioned Speech Inpainting

COIN++: Neural Compression Across Modalities

Real time spectrogram inversion on mobile phone

Contact Info

Product

Resources

About