Few-Shot Learning of New Sound Classes for Target Sound Extraction

Abstract

Target sound extraction consists of extracting the sound of a target acoustic event (AE) class from a mixture of AE sounds. It can be realized using a neural network that extracts the target sound conditioned on a 1-hot vector that represents the desired AE class. With this approach, embedding vectors associated with the AE classes are directly optimized for the extraction of sound classes seen during training. However, it is not easy to extend this framework to new AE classes, i.e. unseen during training. Recently, speech, music, or AE sound extraction based on enrollment audio of the desired sound offers the potential of extracting any target sound in a mixture given only a short audio signal of a similar sound. In this work, we propose combining 1-hot-and enrollment-based target sound extraction, allowing optimal performance for seen AE classes and simple extension to new classes. In experiments with synthesized sound mixtures generated with the Freesound Dataset (FSD) datasets, we demonstrate the benefit of the combined framework for both seen and new AE classes. Besides, we also propose adapting the embedding vectors obtained from a few enrollment audio samples (few-shot) to further improve performance on new classes.

SoundBeam: Target sound extraction conditioned on sound-class labels and enrollment clues for increased performance and continuous learning

“…With the second approach, the target SE embeddings are derived from an enrollment sample or audio query, which is a short recording of an isolated sound similar to the target SE [12], [13], [17]- [19]. Enrollment-based TSE systems learn to extract sounds that share similar characteristics to the enrollment without explicitly relying on SE class labels.…”

“…The initial ideas of SoundBeam were presented in our previous work [13]. In this paper, we provide a more detailed explanation of the approach and an extensive evaluation of the TSE frameworks, covering the following four practical aspects.…”

“…We propose a few-shot adaptation approach, which allows SoundBeam to learn optimal SE embeddings from new SE classes using a few enrollment samples. We introduced this adaption scheme earlier [13], but here we provide a deeper experimental evaluation, including a class-wise analysis of the effect of adaptation 1 SoundBeam generalizes the target speech extraction framework called SpeakerBeam [17] to arbitrary sounds. "Beam" refers to focusing on a particular sound in a mixture but does not imply using a microphone-array or a beamformer [20].…”