Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings

Cramer, Jason; Wu, Ho-Hsiang; Salamon, Justin; Bello, Juan Pablo

doi:10.1109/icassp.2019.8682475

Cited by 228 publications

(185 citation statements)

References 18 publications

Supporting

Mentioning

184

Contrasting

Order By: Relevance

“…Unsupervised learning has undergone major advances with the development of so-called self-supervised learning methods, which define application-specific proxy tasks to encourage neural networks to produce semantically structured representations. We propose a general unimodal and cross-modal representation learning technique based on the proxy task of coincidence prediction, which unifies recent work in audio-only [11] and audio-visual [12,13] self-supervised learning. The goal is to learn separate audio and image embeddings that can predict whether each sound-sound pair or each sound-image pair occurs within some prescribed temporal proximity in which semantic constituents are generally stable.…”

Section: Introductionmentioning

confidence: 92%

Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

Jansen

Ellis

Hershey

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Humans do not acquire perceptual abilities in the way we train machines. While machine learning algorithms typically operate on large collections of randomly-chosen, explicitly-labeled examples, human acquisition relies more heavily on multimodal unsupervised learning (as infants) and active learning (as children). With this motivation, we present a learning framework for sound representation and recognition that combines (i) a self-supervised objective based on a general notion of unimodal and cross-modal coincidence, (ii) a clustering objective that reflects our need to impose categorical structure on our experiences, and (iii) a cluster-based active learning procedure that solicits targeted weak supervision to consolidate categories into relevant semantic classes. By training a combined sound embedding/clustering/classification network according to these criteria, we achieve a new state-of-the-art unsupervised audio representation and demonstrate up to a 20-fold reduction in the number of labels required to reach a desired classification performance.

show abstract

Section: Introductionmentioning

confidence: 92%

Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

Jansen

Ellis

Hershey

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…If none of the outputs corresponding to the KK classes is above the threshold, the sample will be rejected. This MLP architecture is inspired in previous works, such as [15] or the baseline method used in [22]. This also allows for emphasizing the contribution of the proposed autoencoders, verifying their validity in a clearer way.…”

Section: Multi-layer Perceptronmentioning

confidence: 93%

“…This also allows for emphasizing the contribution of the proposed autoencoders, verifying their validity in a clearer way. In this context, the baseline method proposed in [22] uses audio embeddings obtained from the L3net network [15]. As a result, while the baseline employs transfer learning (the method relies on prior knowledge from a pre-trained network), we only make use of the samples available in the training dataset.…”

Section: Multi-layer Perceptronmentioning

confidence: 99%

“…On the one hand, the transfer learning (TL) approach [14] tries to solve the problem of having only few samples by using prior knowledge. This prior-knowledge is usually represented by the use of a neural network pre-trained on external data that is employed as a feature extractor [15]. The other approach lies on novel neural network architectures such as Siamese [16,17], facenet (trained with triplets) [18,19] or on classical networks trained with novel loss functions such as ring loss [20] or center loss [21].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Open Set Audio Classification Using Autoencoders Trained on Few Data

Naranjo-Alcazar

Perez-Castanos²,

Zuccarello³

et al. 2020

Sensors

View full text Add to dashboard Cite

Open-set recognition (OSR) is a challenging machine learning problem that appears when classifiers are faced with test instances from classes not seen during training. It can be summarized as the problem of correctly identifying instances from a known class (seen during training) while rejecting any unknown or unwanted samples (those belonging to unseen classes). Another problem arising in practical scenarios is few-shot learning (FSL), which appears when there is no availability of a large number of positive samples for training a recognition system. Taking these two limitations into account, a new dataset for OSR and FSL for audio data was recently released to promote research on solutions aimed at addressing both limitations. This paper proposes an audio OSR/FSL system divided into three steps: a high-level audio representation, feature embedding using two different autoencoder architectures and a multi-layer perceptron (MLP) trained on latent space representations to detect known classes and reject unwanted ones. An extensive set of experiments is carried out considering multiple combinations of openness factors (OSR condition) and number of shots (FSL condition), showing the validity of the proposed approach and confirming superior performance with respect to a baseline system based on transfer learning.

show abstract

“…Multiple attempts have been made in the particular domain of ambient sound either relying only on audio [4][5][6] or co-training with visualization [7]. However, these works are unsupervised or exploit only a very limited amount of labeled data.…”

Section: Introductionmentioning

confidence: 99%

Limitations of Weak Labels for Embedding and Tagging

Turpault

Serizel

2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

While many datasets and approaches in ambient sound analysis use weakly labeled data, the impact of weak labels on the performance in comparison to strong labels remains unclear. Indeed, weakly labeled data is usually used because it is too expensive to annotate every data with a strong label and for some use cases strong labels are not sure to give better results. Moreover, weak labels are usually mixed with various other challenges like multilabels, unbalanced classes, overlapping events. In this paper, we formulate a supervised problem which involves weak labels. We create a dataset that focuses on difference between strong and weak labels. We investigate the impact of weak labels when training an embedding or an end-to-end classifier. Different experimental scenarios are discussed to give insights into which type of applications are most sensitive to weakly labeled data.

show abstract

Look, Listen, and Learn More: Design Choices for Deep Audio Embeddings

Cited by 228 publications

References 18 publications

Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

Open Set Audio Classification Using Autoencoders Trained on Few Data

Limitations of Weak Labels for Embedding and Tagging

Contact Info

Product

Resources

About