One-Shot Conditional Audio Filtering of Arbitrary Sounds

Abstract

We consider the problem of separating a particular sound source from a single-channel mixture, based on only a short sample of the target source (from the same recording). Using SoundFilter, a waveto-wave neural network architecture, we can train a model without using any sound class labels. Using a conditioning encoder model which is learned jointly with the source separation network, the trained model can be "configured" to filter arbitrary sound sources, even ones that it has not seen during training. Evaluated on the FSD50k dataset, our model obtains an SI-SDR improvement of 9.6 dB for mixtures of two sounds. When trained on Librispeech, our model achieves an SI-SDR improvement of 14.0 dB when separating one voice from a mixture of two speakers. Moreover, we show that the representation learned by the conditioning encoder clusters acoustically similar sounds together in the embedding space, even though it is trained without using any labels.

Few-Shot Learning of New Sound Classes for Target Sound Extraction

“…There are different ways to exploit the embedding vector within the extraction network [2,8,13]. We use here an elementwise multiplication ( operator in Fig.…”

“…The embedding encoder computes the embedding vector e. There are two ways of estimating the embedding vector: using a 1-hot vector [7,9] or an enrollment audio sample [8]. We describe these two approaches in the following subsections.…”

“…The embedding-based approach does not directly optimize the embedding vector for the AE classes, which can result in lower performance for seen AE classes. However, the method can naturally handle new AE classes, when we provide an enrollment sample with similar sound characteristics as the target, and if the system has been trained with a sufficient variety of AE sounds [8].…”

“…It has been a long-standing goal of researchers to reproduce human listening capabilities. Recently, neural network-based target sound extraction has received increased interest as a promising approach towards this goal, with methods developed to extract speech of a target speaker [1][2][3][4], music instruments [5,6] or AE sounds [7][8][9]. In this paper, we focus on the AE sound extraction problem, which is particularly challenging given the large variety of sounds it covers (e.g.…”

“…It uses an extraction neural network that estimates the target AE sound given the sound mixture and an embedding vector that represents the characteristics of the target sound. The embedding vector can be obtained using an embedding encoder that receives either (1) an enrollment audio sample that is similar to the target AE sound [8] or, (2) a 1-hot vector that represents the target AE class [7,9]. The extraction neural network and the embedding encoders are jointly trained.…”