SpeechPainter: Text-conditioned Speech Inpainting

Borsos, Zalán; Sharifi, Matthew; Tagliasacchi, Marco

doi:10.21437/interspeech.2022-194

Cited by 13 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also worth mentioning are other recent works that have applied multi-modal side information as a conditioner for the inpainting algorithm, including video frames [36], symbolic music [37,19], or text [38,39]. Although this idea falls outside the scope of this paper, exploiting multi-modal information may turn out to be beneficial to inpaint large gaps, where the context of the gap does not contain enough information to reconstruct the missing segment.…”

Section: Deep-learning-based Audio Inpaintingmentioning

confidence: 99%

Diffusion-Based Audio Inpainting

Moliner,

Välimäki

2024

J. Audio Eng. Soc.

View full text Add to dashboard Cite

Audio inpainting aims to reconstruct missing segments in corrupted recordings. Most existing methods produce plausible reconstructions when the gap lengths are short but struggle to reconstruct gaps larger than about 100 ms. This paper explores diffusion models, a recent class of deep learning models, for the task of audio inpainting. The proposed method uses an unconditionally trained generative model, which can be conditioned in a zero-shot fashion for audio inpainting and is able to regenerate gaps of any size. An improved deep neural network architecture based on the constant-Q transform that allows the model to exploit pitch-equivariant symmetries in audio is also presented. The performance of the proposed algorithm is evaluated through objective and subjective metrics for the task of reconstructing short to mid-sized gaps, up to 300 ms. The results of a formal listening test indicate that, for short gaps in the range of 50 ms, the proposed method delivers performance comparable to the baselines. For wider gaps up to 300 ms long, the authors' method outperforms the baselines and retains good or fair audio quality. The method presented in this paper can be applied to restoring sound recordings that suffer from severe local disturbances or dropouts.

show abstract

Section: Deep-learning-based Audio Inpaintingmentioning

confidence: 99%

Diffusion-Based Audio Inpainting

Moliner,

Välimäki

2024

J. Audio Eng. Soc.

View full text Add to dashboard Cite

show abstract

“…While earlier approches to musical audio generation were limited in terms of producing high quality outputs (Dhariwal et al, 2020) or semantically consistent long audios (Hawthorne et al, 2022), recent research has achieved a level of quality that allows for an enjoyable listening experience. A first line of work casts the task of music generation as categorical prediction in the discrete token space provided by a neural audio codec Zeghidour et al, 2022), and trains a Transformer-based (Vaswani et al, 2017) model for next token prediction (Borsos et al, 2023a) or parallel token decoding (Borsos et al, 2023b;Garcia et al, 2023;Parker et al, 2024).…”

Section: Related Workmentioning

confidence: 99%

“…We address this with a hierarchical transformer, similarly to Lee et al (2022); Yang et al (2023); . Finally, instead of the original autoregressive fine acoustic modelling stage of MusicLM, we use Soundstorm (Borsos et al, 2023b) for achieving efficient parallel generation.…”

Section: Musiclmmentioning

confidence: 99%

HIGhER: Improving instruction following with Hindsight Generation for Experience Replay

Cideron

Seurin

Strub³

et al. 2020

2020 IEEE Symposium Series on Computational Intelligence (SSCI)

View full text Add to dashboard Cite

Language creates a compact representation of the world and allows the description of unlimited situations and objectives through compositionality. While these characterizations may foster instructing, conditioning or structuring interactive agent behavior, it remains an open-problem to correctly relate language understanding and reinforcement learning in even simple instruction following scenarios. This joint learning problem is alleviated through expert demonstrations, auxiliary losses, or neural inductive biases. In this paper, we propose an orthogonal approach called Hindsight Generation for Experience Replay (HIGhER) that extends the Hindsight Experience Replay approach to the language-conditioned policy setting. Whenever the agent does not fulfill its instruction, HIGhER learns to output a new directive that matches the agent trajectory, and it relabels the episode with a positive reward. To do so, HIGhER learns to map a state into an instruction by using past successful trajectories, which removes the need to have external expert interventions to relabel episodes as in vanilla HER. We show the efficiency of our approach in the BabyAI environment, and demonstrate how it complements other instruction following methods.

show abstract

“…Bai et al [9] suggests an alignment-aware acoustic and text pretraining method, which can be directly applied to speech editing by reconstructing masked acoustic signals through text input and acoustic text alignment. What's more, SpeechPainter [10] leverages an auxiliary textual input to fill in gaps of up to one second in speech samples and generalize it to unseen speakers. However, when applied to speech editing, all the existing methods [6,7,8,9] based on neural networks do partial inference instead of entire inference, as shown in Figure 1(a).…”

Section: Introductionmentioning

confidence: 99%

Cross-utterance Conditioned Coherent Speech Editing

Yu¹,

Li²,

Zu³

et al. 2023

Interspeech 2023

View full text Add to dashboard Cite

Text-based speech editing systems are developed to enable users to modify speech based on the transcript. Existing state-ofthe-art editing systems based on neural networks do partial inferences with no exception, that is, only generate new words that need to be replaced or inserted. This manner usually leads to the prosody of the edited part being inconsistent with the surrounding speech and a failure to handle the alteration of intonation. To address these problems, we propose a cross-utterance conditioned coherent speech editing system, that first does the entire reasoning at the inference time. Our proposed system can generate speech by utilizing speaker information, context, acoustic features, and the mel-spectrogram from the original audio. Experiments conducted on subjective and objective metrics demonstrate that our approach outperforms the baseline on various editing operations regarding naturalness and prosody consistency.

show abstract

SpeechPainter: Text-conditioned Speech Inpainting

Cited by 13 publications

References 17 publications

Diffusion-Based Audio Inpainting

Diffusion-Based Audio Inpainting

HIGhER: Improving instruction following with Hindsight Generation for Experience Replay

Cross-utterance Conditioned Coherent Speech Editing

Contact Info

Product

Resources

About