A Versatile Diffusion-based Generative Refiner for Speech Enhancement

Abstract

Although deep neural network (DNN)-based speech enhancement (SE) methods outperform the previous non-DNN-based ones, they often degrade the perceptual quality of generated outputs. To tackle this problem, We introduce a DNN-based generative refiner aiming to improve perceptual speech quality pre-processed by an SE method. As the refiner, we train a diffusion-based generative model by utilizing a dataset consisting of clean speech only. Then, the model replaces the degraded and distorted parts caused by a preceding SE method with newly generated clean parts by denoising diffusion restoration. Once our refiner is trained on a set of clean speech, it can be applied to various SE methods without additional training specialized for each SE module. Therefore, our refiner can be a versatile post-processing module w.r.t. SE methods and has high potential in terms of modularity. Experimental results show that our method improved perceptual speech quality regardless of the preceding SE methods used.

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

“…Research on speech enhancement has achieved significant improvement in terms of signal-to-noise (SNR) ratio but sometimes degrades the speech quality (e.g., naturalness), leading to the degradation of downstream applications. To remove the distortions of speech enhancement outputs, Refiner [93] applies a diffusion model pretrained on clean speech data to detect the degraded part, and then replaces them with newly generated clean ones in the manner of denoising diffusion restoration models(DDRM) [35]. Experimental results show that Refiner [93] is versatile since it improves speech quality with regard to various speech enhancement methods.…”

“…To remove the distortions of speech enhancement outputs, Refiner [93] applies a diffusion model pretrained on clean speech data to detect the degraded part, and then replaces them with newly generated clean ones in the manner of denoising diffusion restoration models(DDRM) [35]. Experimental results show that Refiner [93] is versatile since it improves speech quality with regard to various speech enhancement methods. Moreover, the Refiner [93] can also be integrated into the speech enhancement model for joint optimization in the future.…”

“…Experimental results show that Refiner [93] is versatile since it improves speech quality with regard to various speech enhancement methods. Moreover, the Refiner [93] can also be integrated into the speech enhancement model for joint optimization in the future. Audio super-resolution [46,56], also widely known as upsampling [84] or bandwidth extension [46], aims to generate audio of a high sampling rate from that of a low sampling rate via extending its bandwidth.…”