VoiceFixer: Toward General Speech Restoration with Neural Vocoder

Liu, Haohe; Kong, Qiuqiang; Tian, Qiao; Zhao, Yan; Wang, DeLiang; Huang, Chuanzeng; Wang, Yuxuan

doi:10.48550/arxiv.2109.13731

Cited by 7 publications

(8 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The estimated speech signal obtained may be distorted due to oversubtraction. We will then use a pre-trained VoiceFixer model [17], which shows great performance in restoring strongly degraded human speech, to reconstruct this. Another crucial factor in Eq.6 is the matching 𝑡 between the robot speech and the reference speech.…”

Section: Audiomentioning

confidence: 99%

Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction

Li,

Hindriks,

Kunneman

2024

Proceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction

View full text Add to dashboard Cite

In this paper, we study how well human speech can automatically be filtered when this overlaps with the voice and fan noise of a social robot, Pepper. We ultimately aim for an HRI scenario where the microphone can remain open when the robot is speaking, enabling a more natural turn-taking scheme where the human can interrupt the robot. To respond appropriately, the robot would need to understand what the interlocutor said in the overlapping part of the speech, which can be accomplished by target speech extraction (TSE). To investigate how well TSE can be accomplished in the context of the popular social robot Pepper, we set out to manufacture a datase composed of a mixture of recorded speech of Pepper itself, its fan noise (which is close to the microphones), and human speech as recorded by the Pepper microphone, in a room with low reverberation and high reverberation. Comparing a signal processing approach, with and without post-filtering, and a convolutional recurrent neural network (CRNN) approach to a state-of-the-art speaker identification-based TSE model, we found that the signal processing approach without post-filtering yielded the best performance in terms of Word Error Rate on the overlapping speech signals with low reverberation, while the CRNN approach is more robust for reverberation. Moreover, the best performance is not sufficient for consistent comprehension after filtering, while we see a large diversity in performance across our dataset. We conclude that, first, the human speech volume and pitch strongly affect the performance of the proposed method's results; second, the signal processing method based on speech masking and spectral subtraction is keen to reverberation, while the neural network method is robust; third, the batch normalization layer in TSE models is not useful for filtering the interference speech when it is significantly more powerful than the target speech. These results show that estimating the human voice in overlapping speech with a robot is possible in real-life application, provided that the room reverberation is low and the human speech has a high volume or high pitch. CCS CONCEPTS• Human-centered computing → Sound-based input / output.

show abstract

Section: Audiomentioning

confidence: 99%

Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction

Li,

Hindriks,

Kunneman

2024

Proceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction

View full text Add to dashboard Cite

show abstract

“…NU-Wave 2 [23] further improves NU-Wave [49] from two aspects. On the one hand, NU-Wave 2 [23] adopts short-time Fourier convolution (STFC) to overcome the limitations of NU-Wave [49] that fails to generate harmonics of vowels [49] and various frequency bands [57,72]. On the other hand, different from prior models that the initial and target sampling rates are fixed, NU-Wave 2 [23] defines a new task general neural audio upsampling that the inputs can be any sampling rate for a single model.…”

Section: Unsupervised Restorationmentioning

confidence: 99%

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

Zhang¹,

Zhang²,

Zheng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction.With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

show abstract

“…where F and F −1 refer to the Fourier transform and its inverse operation, respectively,H ϕ j is a zero-phase frequency-domain filter computed through (10) using the parameters in ϕ j , and ⊙ is the Hadamard product, or element-wise multiplication. This operation is, in practice, realized in a frame-by-frame manner using a short-time Fourier transform, using a Hamming window length of 4096 samples and a hop size of 2048 samples.…”

Section: Joint Posterior Sampling and Filter Inferencementioning

confidence: 99%

BEHM-GAN: Bandwidth Extension of Historical Music Using Generative Adversarial Networks

Moliner

Välimäki

2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

Audio bandwidth extension aims to expand the spectrum of bandlimited audio signals. Although this topic has been broadly studied during recent years, the particular problem of extending the bandwidth of historical music recordings remains an open challenge. This paper proposes a method for the bandwidth extension of historical music using generative adversarial networks (BEHM-GAN) as a practical solution to this problem. The proposed method works with the complex spectrogram representation of audio and, thanks to a dedicated regularization strategy, can effectively extend the bandwidth of out-of-distribution real historical recordings. The BEHM-GAN is designed to be applied as a second step after denoising the recording to suppress any additive disturbances, such as clicks and background noise. We train and evaluate the method using solo piano classical music. The proposed method outperforms the compared baselines in both objective and subjective experiments. The results of a formal blind listening test show that BEHM-GAN significantly increases the perceptual sound quality in early-20th-century gramophone recordings. For several items, there is a substantial improvement in the mean opinion score after enhancing historical recordings with the proposed bandwidthextension algorithm. This study represents a relevant step toward data-driven music restoration in real-world scenarios.

show abstract

VoiceFixer: Toward General Speech Restoration with Neural Vocoder

Cited by 7 publications

References 58 publications

Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction

Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction

Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI

BEHM-GAN: Bandwidth Extension of Historical Music Using Generative Adversarial Networks

Contact Info

Product

Resources

About