Phase Reconstruction for Time-Frequency Inpainting

Kreme, Ama Marina; Emiya, Valentin; Chaux, Caroline

doi:10.1007/978-3-319-93764-9_39

Cited by 4 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, PR is of paramount importance for tasks that involve audio signal reconstruction from incomplete timefrequency observations. Consequently, it has attracted some attention for many applications such as speech enhancement [4]- [6], source separation [7]- [10] or audio restoration (e.g., click removal [11] or time-frequency inpainting [12]).…”

Section: Introductionmentioning

confidence: 99%

Phase Retrieval With Bregman Divergences and Application to Audio Signal Recovery

Vial

Magron

Oberlin

et al. 2021

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Phase retrieval (PR) aims to recover a signal from the magnitudes of a set of inner products. This problem arises in many audio signal processing applications which operate on a short-time Fourier transform magnitude or power spectrogram, and discard the phase information. Recovering the missing phase from the resulting modified spectrogram is indeed necessary in order to synthesize time-domain signals. PR is commonly addressed by considering a minimization problem involving a quadratic loss function. In this paper, we adopt a different standpoint. Indeed, the quadratic loss does not properly account for some perceptual properties of audio, and alternative discrepancy measures such as beta-divergences have been preferred in many settings. Therefore, we formulate PR as a new minimization problem involving Bregman divergences. Since these divergences are not symmetric with respect to their two input arguments in general, they lead to two different formulations of the problem. To optimize the resulting objective, we derive two algorithms based on accelerated gradient descent and alternating direction method of multipliers. Experiments conducted on audio signal recovery from spectrograms that are either exact or estimated from noisy observations highlight the potential of our proposed methods for audio restoration. In particular, leveraging some of these Bregman divergences induce better performance than the quadratic loss when performing PR from spectrograms under very noisy conditions.

show abstract

Section: Introductionmentioning

confidence: 99%

Phase Retrieval With Bregman Divergences and Application to Audio Signal Recovery

Vial

Magron

Oberlin

et al. 2021

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

show abstract

“…Our base model architecture is inspired from [15]. There have been several studies on inpainting, particularly on speech, that strengthen our concept of context information with inpainting [16,17,18,19,20,21,22,23].…”

Section: Introductionmentioning

confidence: 99%

Correcting Mispronunciations in Speech using Spectrogram Inpainting

Ben-Simon¹,

Kreuk²,

Awwad³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning a new language involves constantly comparing speech productions with reference productions from the environment. Early in speech acquisition, children make articulatory adjustments to match their caregivers' speech. Grownup learners of a language tweak their speech to match the tutor reference. This paper proposes a method to synthetically generate correct pronunciation feedback given incorrect production. Furthermore, our aim is to generate the corrected production while maintaining the speaker's original voice.The system prompts the user to pronounce a phrase. The speech is recorded, and the samples associated with the inaccurate phoneme are masked with zeros. This waveform serves as an input to a speech generator, implemented as a deep learning inpainting system with a U-net architecture, and trained to output a reconstructed speech. The training set is composed of unimpaired proper speech examples, and the generator is trained to reconstruct the original proper speech. We evaluated the performance of our system on phoneme replacement of minimal pair words of English as well as on children with pronunciation disorders. Results suggest that human listeners slightly prefer our generated speech over a smoothed replacement of the inaccurate phoneme with a production of a different speaker.

show abstract

“…Therefore, PR is of paramount importance for tasks that involve audio signal reconstruction from incomplete time-frequency observations. Consequently, it has attracted some attention for many applications such as speech enhancement [4,5,6], source separation [7,8,9,10] or audio restoration (e.g., click removal [11] or time-frequency inpainting [12]).…”

Section: Introductionmentioning

confidence: 99%

Phase retrieval with Bregman divergences and application to audio signal recovery

Vial,

Magron,

Oberlin

et al. 2020

Preprint

View full text Add to dashboard Cite

Phase retrieval (PR) aims to recover a signal from the magnitudes of a set of inner products. This problem arises in many audio signal processing applications which operate on a short-time Fourier transform magnitude or power spectrogram, and discard the phase information. Recovering the missing phase from the resulting modified spectrogram is indeed necessary in order to synthesize time-domain signals. PR is commonly addressed by considering a minimization problem involving a quadratic loss function. In this paper, we adopt a different standpoint. Indeed, the quadratic loss does not properly account for some perceptual properties of audio, and alternative discrepancy measures such as beta-divergences have been preferred in many settings. Therefore, we formulate PR as a new minimization problem involving Bregman divergences. We consider a general formulation that actually addresses two problems, since it accounts for the non-symmetry of these divergences in general. To optimize the resulting objective, we derive two algorithms based on accelerated gradient descent and alternating direction method of multiplier. Experiments conducted on audio signal recovery from either exact or modified spectrograms highlight the potential of our proposed methods for audio restoration. In particular, leveraging some of these Bregman divergences induce better performance than the quadratic loss when performing PR from highly degraded spectrograms.

show abstract

Phase Reconstruction for Time-Frequency Inpainting

Cited by 4 publications

References 19 publications

Phase Retrieval With Bregman Divergences and Application to Audio Signal Recovery

Phase Retrieval With Bregman Divergences and Application to Audio Signal Recovery

Correcting Mispronunciations in Speech using Spectrogram Inpainting

Phase retrieval with Bregman divergences and application to audio signal recovery

Contact Info

Product

Resources

About