Zero-Shot Audio Source Separation through Query-Based Learning from Weakly-Labeled Data

Chen, Ke; Du, Xingjian; Zhu, Bilei; Ma, Zejun; Berg-Kirkpatrick, Taylor; Dubnov, Shlomo

doi:10.1609/aaai.v36i4.20366

Cited by 18 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methods separate the target sound conditioned on the average audio embedding of a few audio examples of the target source, which requires labeled single sources for training. The work in [15], [16] proposes to train the audio-queried sound separation system with large-scale weakly labeled data (e.g., AudioSet [30]), which is achieved by first using a sound event detection model [31] to detect the anchor segment of sound events which are further used to constitute the artificial mixtures for the training of audio-queried sound separation models. These audio-queried sound separation approaches have shown great potential in the separation of unseen sound sources.…”

Section: B Query-based Sound Separationmentioning

confidence: 99%

Separate What You Describe: Language-Queried Audio Source Separation

Liu¹,

Liu²,

Kong³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given a natural language query, which provides a natural and scalable interface for digital audio applications. Recent works on LASS, despite attaining promising separation performance on specific sources (e.g., musical instruments, limited classes of audio events), are unable to separate audio concepts in the open domain. In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries. We train AudioSep on large-scale multimodal datasets and extensively evaluate its capabilities on numerous tasks including audio event separation, musical instrument separation, and speech enhancement. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability using audio captions or text labels as queries, substantially outperforming previous audio-queried and language-queried sound separation models. For reproducibility of this work, we will release the source code, evaluation benchmark and pre-trained model at: https://github.com/Audio-AGI/AudioSep.

show abstract

Section: B Query-based Sound Separationmentioning

confidence: 99%

Separate What You Describe: Language-Queried Audio Source Separation

Liu¹,

Liu²,

Kong³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

“…Several works have utilized VAE models for exploring audio representations, including VAE models for finding disentangled audio representations (Luo, Agres, & Herremans, 2019) and VAE models for modeling audio containing speech (Hsu, Zhang, & Glass, 2017). Apart from VAE, simple convolutional models are used in different music tasks such as music recommendation (Chen, Liang, Ma, & Gu, 2021) and source separation (Chen, Du, et al, 2021). In our experiments for audio representations, we chose to use a convolutional VAE to map the short-time Fourier transform (STFT) representation of audio into a lower dimensional latent representation.…”

Section: Vae For Audiomentioning

confidence: 99%

Deep Music Information Dynamics Novel Framework for Reduced Neural-Network Music Representation with Applications to Midi and Audio Analysis and Improvisation

Dubnov¹,

Chen²,

Huang³

2022

Journal of Creative Music Systems

Self Cite

View full text Add to dashboard Cite

Generative musical models often comprise of multiple levels of structure, presuming that the process of composition moves between background to foreground, or between generating musical surface and some deeper and reduced representation that governs hidden or latent dimensions of music.  In this paper we are using a recently proposed framework called Deep Musical Information Dynamics (DMID) to explore information contents of deep neural models of music through rate reduction of latent representation streams, which is contrasted with hight rate information dynamics of the musical surface. This approach is partially motivated by rate-distortion theories of human cognition, providing a framework for exploring possible relations between imaginary anticipations existing in the listener's or composer's mind, and the information dynamics of the sensory (acoustic) or symbolic score data. In the paper the DMID framework is demonstrated using several experiments with symbolic (MIDI) and acoustic (spectral) music representations. We use variational encoding to learn a latent representation of the musical surface. This embedding is further reduced using a bit-allocation method into a second stream of low bit-rate encoding. The combined loss includes temporal information in terms of predictive properties for each encoding stream, and accuracy loss measured in terms of mutual information between the encoding at low rate and the high rate surface representations. For the case of counterpoint, we also study the mutual information between two voices in a musical piece at different levels of information reduction.The DMID framework allows to explore aspects of computational creativity in terms of juxtaposition of latent/imaginary surprisal aspects of deeper structure with music surprisal on the surface level, done in a manner that is quantifiable and computationally tractable. The relevant information theory modeling and analysis methods are discussed in the paper, suggesting that a trade off between compression and prediction play an important factor in the analysis and design of creative musical systems.

show abstract

“…Unlike traditional MSS methods, which mainly focus on the acoustic characteristics of sound sources for studying songs * School of Music, Henan Vocational Institute of Arts, Zhengzhou, 451464, China (unwenzhy1234@163.com) and instruments in mixed audio, in recent years, the growth of big data and computing technology has led to the introduction of more modern computing technologies into the field of audio processing, and has also brought new ideas to MSS [5][6]. Chen et al addressed the problem of poor separation performance of multi-source mixed audio sources by utilizing three sets of channels to train the audio set based on deep learning, thereby improving separation performance and expanding its generalization [7]. Huang et al proposed a compatible prediction model based on self supervised and semi supervised learning to address the issue of low prediction performance of audio element compatibility in current mixed audio MSS.…”

Section: Introductionmentioning

confidence: 99%

Application of Unet-SE-Bisru Algorithm for Music Signal Processing in Music Source Separation

Zhang

2024

SCPE

View full text Add to dashboard Cite

At present, the use of time-domain deep learning for end-to-end neural network models has the problem of long training time and poor performance in music source separation. To address this issue, a U-network squeezing excitation bidirectional simple recursive unit model was proposed based on the deep extractor model. Replace Unet SE Bisru with Unet SE Bisru in the following text. This model improves the bidirectional long short-term memory network into a bidirectional simple recurrent unit, and then introduces attention mechanisms in the generalized encoding and decoding layers. The squeezing excitation block is used to selectively extract features based on the type of audio to be separated. Finally, group normalization is added after onedimensional convolution, And its effectiveness was verified. The experimental results show that the signal noise distortion ratio in the improved model is 5.68 decibels compared to the bidirectional simple recursive unit value, which is higher than the 5.55 decibels of bidirectional long short-term memory. After adding the squeezing excitation module, the overall increase is about 0.1-0.5 decibels. In addition, in the model comparison, the three indicators of the improved model with the same number of channels were 5.68 decibels, 5.91 decibels, and 11.28 decibels, respectively, higher than the benchmark model. Compared with other music source classification models, the improved model has better comprehensive separation performance. Although some indicators are lower than the comparison model, the signal noise distortion ratio of drum and bass is 6.11 decibels and 6.36 decibels, which is better than the comparison model. Overall, the improved model has high performance in music source separation for music signal processing and can be effectively applied in practical music source separation.

show abstract

Zero-Shot Audio Source Separation through Query-Based Learning from Weakly-Labeled Data

Cited by 18 publications

References 15 publications

Separate What You Describe: Language-Queried Audio Source Separation

Separate What You Describe: Language-Queried Audio Source Separation

Deep Music Information Dynamics Novel Framework for Reduced Neural-Network Music Representation with Applications to Midi and Audio Analysis and Improvisation

Application of Unet-SE-Bisru Algorithm for Music Signal Processing in Music Source Separation

Contact Info

Product

Resources

About