DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learning

Abstract

Inspired by the recent progress in self-supervised learning for computer vision, in this paper, through the DeLoRes learning framework, we introduce two new general-purpose audio representation learning approaches, the DeLoRes-S and DeLoRes-M. Our main objective is to make our network learn representations in a resource-constrained setting (both data and compute), that can generalize well across a diverse set of downstream tasks. Inspired from the Barlow Twins objective function, we propose to learn embeddings that are invariant to distortions of an input audio sample, while making sure that they contain non-redundant information about the sample. To achieve this, we measure the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of an audio segment sampled from an audio file and make it as close to the identity matrix as possible. We call this the DeLoRes learning framework, which we employ in different fashions with the DeLoRes-S and DeLoRes-M. We use a combination of a small subset of the largescale AudioSet dataset and FSD50K for self-supervised learning and are able to learn with less than half the parameters compared to state-of-the-art algorithms. For evaluation, we transfer these learned representations to 11 downstream classification tasks, including speech, music, and animal sounds, and achieve stateof-the-art results on 7 out of 11 tasks on linear evaluation with DeLoRes-M and show competitive results with DeLoRes-S, even when pre-trained using only a fraction of the total data when compared to prior art. Our transfer learning evaluation setup also shows extremely competitive results for both DeLoRes-S and DeLoRes-M, with DeLoRes-M achieving state-of-the-art in 4 tasks. In addition to being simple and intuitive, our pre-training procedure is amenable to compute through its inherent nature of construction and does not require careful implementation details to avoid trivial or degenerate solutions like prior art in this domain. Furthermore, we conduct extensive ablation studies on our training algorithm, model architecture and results, and make all our code and pre-trained models publicly available 1 .

WaveBYOL: Self-Supervised Learning for Audio Representation From Raw Waveforms

“…We propose an augmentation layer for generating various views from raw waveforms and an encoding layer for learning meaningful representations. Although performance differences are observed depending on the downstream tasks, WaveBYOL generally shows competitive performance compared to the previously proposed models [12], [14], [15], [17], [18], [19].…”

“…In frozen-model evaluation, a linear classifier with a multilayer perceptron (MLP) layer is trained to classify a new dataset based on top of the frozen pretrained network, and in fine-tuning, we allow all weights to vary during training. In the frozen-model evaluation experiment, WaveBYOL is compared with COLA [12], DeLoRes [15], BYOL-A [17], [18], and ATST [19], and in the fine-tuning experiment, it is compared with COLA, DeLoRes, SSAST [14], and ATST.…”

“…Self-supervised audio spectrogram transformer (SSAST) [13], [14] is a transformer-based self-supervised learning model, and the authors proposed a masked-spectrogram patch modeling technique. Decorrelating latent spaces for low-resource audio representation learning (DeLoRes) [15] is a framework consisting of two simple self-supervised pretraining methodologies for learning general-purpose audio representations of speech and sound. Inspired by the Barlow Twins framework [16], the authors used a redundancy reduction-based loss function to make the computed cross-correlation matrix as close to the identity matrix as possible regarding the embeddings of the augmented sample pairs of the same audio segments.…”

LABERT: A Combination of Local Aggregation and Self-Supervised Speech Representation Learning for Detecting Informative Hidden Units in Low-Resource ASR Systems

“…BEST-RQ [9] masks the speech input and feeds the masks into an encoder to learn masked parts of speech based on the unmasked part through random-projection quantizers. DeLoRes [10] learns general purpose audio representations through an invariance and redundancy reduction based objective function. wav2vec 2.0 [3] enhances vq-wav2vec through a single-stage training by masking the input speech data into the latent space and then solves a contrastive task defined over a quantization of the latent representations by computing the similarity between the predicted masked vectors and original vectors.…”