Bandwidth Extension is All You Need

Abstract

Speech generation and enhancement have seen recent breakthroughs in quality thanks to deep learning. These methods typically operate at a limited sampling rate of 16-22kHz due to computational complexity and available datasets. This limitation imposes a gap between the output of such methods and that of high-fidelity (≥44kHz) real-world audio applications. This paper proposes a new bandwidth extension (BWE) method that expands 8-16kHz speech signals to 48kHz. The method is based on a feed-forward WaveNet architecture trained with a GAN-based deep feature loss. A mean-opinionscore (MOS) experiment shows significant improvement in quality over state-of-the-art BWE methods. An AB test reveals that our 16to-48kHz BWE is able to achieve fidelity that is typically indistinguishable from real high-fidelity recordings. We use our method to enhance the output of recent speech generation and denoising methods, and experiments demonstrate significant improvement in sound quality over these baselines. We propose this as a general approach to narrow the gap between generated speech and recorded speech, without the need to adapt such methods to higher sampling rates.

BEHM-GAN: Bandwidth Extension of Historical Music using Generative Adversarial Networks

“…Although only a few studies have applied GAN models to bandwidth extension of music signals [10], [39], many recent works have applied them for speech [14], [13], [40]. Eskimez et al [40] proposed one of the earliest works using an adversarial approach for speech super-resolution.…”

“…Moreover, timedomain discriminators help keep the quasi-periodic structure of raw audio, generating a more naturally sounding harmonic series. We also experimented incorporating STFT-based discriminators, following the methodology suggested by Su et al [13], but we noticed a considerable decline in the generated audio quality. The STFT discriminators force the generator to inpaint harmonic-like shapes in the spectrogram without taking into account their coherence with the lower part of the spectrum, thus generating metallic and unrealistic sounds.…”

“…We also experimented with TFiLM [33], the GAN-based HiFi-GAN [13], and the diffusion probabilistic NU-Wave models [15]. However, we did not obtain positive results using these methods for bandwidth-extending historical recordings.…”

“…1) Log-Spectral Distance: The LSD, a frequency-domain metric that has been popularly used in bandwidth-extension literature [11], [12], [13], [40], is defined as:…”

“…During recent years, many works have used modern deep learning technologies for bandwidth extension, but usually with the final goal of increasing the sampling rate of modern digital audio signals. Only a few exceptions are relevant to music signal processing [8], [9], [10], whereas most of these studies focus on processing speech [11], [12], [13], [14], [15].…”