BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Abstract

Pre-trained models are essential as feature extractors in modern machine learning systems in various domains. In this study, we hypothesize that representations effective for general audio tasks should provide multiple aspects of robust features of the input sound. For recognizing sounds regardless of perturbations such as varying pitch or timbre, features should be robust to these perturbations. For serving the diverse needs of tasks such as recognition of emotions or music genres, representations should provide multiple aspects of these robust features, such as local and global features and their statistics. To implement our principle, we propose a self-supervised learning method: Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"). BYOL-A pre-trains representations of the input sound themselves invariant to audio data augmentations by minimizing the difference between a pair of augmented input variants, which makes the learned representations robust to the perturbations of sounds. In the BYOL-A encoder, the global pooling calculates representations to form multi-aspect information by combining statistics of frequency-and channelwise, local, and global features. As a result, the learned representations should provide multi-aspect robust features of the input and serve various needs of diverse tasks. We evaluated general audio task performance among previous state-of-the-art methods, and BYOL-A showed competitive results in all tasks with the best average result of 72.4 %. Besides, BYOL-A sets new records of 57.6 % on VoxCeleb1 and 63.8 % on CREMA-D. We also conducted extensive ablation experiments and validated the contributions of BYOL-A components. Our code is available online.

Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Load

“…On the other hand, hybrid representation models, which aim to take advantage of both data-driven and handcrafted features, yielded significantly better performance. The two proposed hybrid models, one using BYOL-A's CNN encoder [19] and one with CvT encoding [37], consistently outperformed their corresponding BYOL-S models. This result suggests that adding the DSP-based supervision to the self-supervised representation learning framework helped to improve its generalization capacity in cognitive/physical load detection tasks.…”

“…As in [18], we use the features from layer 19 as more effective than other layers of the model. BYOL-A -Unlike contrastive learning ferameorks, Bootstrap Your Own Latent for Audio (BYOL-A) [19] generates audio representations using two augmented views of a single audio sample, inspired by the success of BYOL [36] for image representation. To obtain audio representations, the log-mel spectrogram (LMS) of an input audio sample is first fed to a data augmentation module, yielding two randomly augmented copies of the input LMS (Figure 1).…”

“…Yet, training such models directly on task load corpora remains a challenging issue due to their small sizes, as large or complex networks tend to overfit and have poor generalisation capacity. To overcome this issue, several studies have shown that pre-training models on large corpora of speech or audio can produce state-of-the-art performance in various dataconstrained speech processing and paralinguistic tasks such as speech emotion recognition [17], speaker or language identification [18][19][20].…”

DeLoRes: Decorrelating Latent Spaces for Low-Resource Audio Representation Learning

“…Self-supervised learning in speech and audio aims towards learning representations that contain high-level information from acoustic signals which can be further used in diverse sets of downstream tasks. Model weights learned through selfsupervision are either used as feature extractors under the linear evaluation protocol [27], [33] or used together with transfer learning for end-to-end fine-tuning with an added prediction-head for the downstream task [3], [7], [21]. Features learned through self-supervised speech representation learning have already proven to outperform other low-level features such as filter-banks and mel-frequency cepstral coefficients (MFCCs).…”

“…Moreover, [3] show that the quality of mined negative samples can affect the performance. On the other hand, the method proposed by [33] uses a momentum encoder, where a moving-average network is used to produce prediction targets for optimizing the MSE loss between two batches of augmented samples of the same audio segments. However, the symmetry-breaking network design has been found crucial for this approach.…”