Finding Strength in Weakness: Learning to Separate Sounds with Weak Supervision

Abstract

While there has been much recent progress using deep learning techniques to separate speech and music audio signals, these systems typically require large collections of isolated sources during the training process. When extending audio source separation algorithms to more general domains such as environmental monitoring, it may not be possible to obtain isolated signals for training. Here, we propose objective functions and network architectures that enable training a source separation system with weak labels. In this scenario, weak labels are defined in contrast with strong time-frequency (TF) labels such as those obtained from isolated sources, and refer either to frame-level weak labels where one only has access to the time periods when different sources are active in an audio mixture, or to clip-level weak labels that only indicate the presence or absence of sounds in an entire audio clip. We train a separator that estimates a TF mask for each type of sound event, using a sound event classifier as an assessor of the separator's performance to bridge the gap between the TF-level separation and the ground truth weak labels only available at the frame or clip level. Our objective function requires the classifier applied to a separated source to assign high probability to the class corresponding to that source and low probability to all other classes. The objective function also enforces that the separated sources sum up to the mixture. We benchmark the performance of our algorithm using synthetic mixtures of overlapping events created from a database of sounds recorded in urban environments. Compared to training a network using isolated sources, our model achieves somewhat lower but still significant SI-SDR improvement, even in scenarios with significant sound event overlap.

Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention

“…However, these experiments relied on having sufficient supervised training data and were evaluated only on test sets with similar environmental conditions and sound distributions. In order to extend the reach of this approach, methods have been proposed to train separation models with no access to ground truth clean sources by utilizing weak class labels [28], the spatial separability of the sources [34,31,8] and self-supervision in the form of MixIT [41]. This makes it possible to learn separation of signals well outside the domains for which isolated source databases exist.…”

Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds

“…To avoid these issues, a number of recent works have proposed replacing the strong supervision of reference source signals with weak supervision labels from related modalities such as sound class [25,18], visual input [8], or spatial location from multi-microphone recordings [31,27,4]. Most recently, [35] proposed mixture invariant training (MixIT), which provides a purely unsupervised source separation framework for a variable number of latent sources.…”

Unsupervised Sound Separation Using Mixture Invariant Training

“…To that end, weakly supervised training has been proposed to substitute the strong labels of source references with another modality such as class labels, visual features, or spatial information. In [32] class labels were used as a substitute for signal-level losses. The spatial locations of individual sources, which can be inferred from multichannel audio, has also been used to guide learning of single-channel separation [38,35,8].…”