BYOL for Audio: Exploring Pre-Trained General-Purpose Audio Representations

Abstract

Pre-trained models are essential as feature extractors in modern machine learning systems in various domains. In this study, we hypothesize that representations effective for general audio tasks should provide multiple aspects of robust features of the input sound. For recognizing sounds regardless of perturbations such as varying pitch or timbre, features should be robust to these perturbations. For serving the diverse needs of tasks such as recognition of emotions or music genres, representations should provide multiple aspects of information, such as local and global features. To implement our principle, we propose a self-supervised learning method: Bootstrap Your Own Latent (BYOL) for Audio (BYOL-A, pronounced "viola"). BYOL-A pretrains representations of the input sound invariant to audio data augmentations, which makes the learned representations robust to the perturbations of sounds. Whereas the BYOL-A encoder combines local and global features and calculates their statistics to make the representation provide multi-aspect information. As a result, the learned representations should provide robust and multi-aspect information to serve various needs of diverse tasks. We evaluated the general audio task performance of BYOL-A compared to previous state-of-the-art methods, and BYOL-A demonstrated generalizability with the best average result of 72.4% and the best VoxCeleb1 result of 57.6%. Extensive ablation experiments revealed that the BYOL-A encoder architecture contributes to most performance, and the final critical portion resorts to the BYOL framework and BYOL-A augmentations. Our code is available online for future studies.

ATST: Audio Representation Learning with Teacher-Student Transformer

“…1) Linear Evaluation Results: Table IV shows the linear evaluation results on six tasks. For a fair comparison, we compare with other methods that also use Audioset for pretraining and have also reported the linear evaluation results in their papers, including TRILL [45], COLA [3], BYOL-A [5], BYOL-A-v2 [11], SF NFNET-F0 [46] and M2D [15]. The proposed ATST-Clip is developed based on BYOL-A and BYOL-A-V2, using a transformer encoder and a new view creation strategy.…”

“…C. Results on Frame-level Downstream Task -Sound Event Detection 1) Comparison Methods: We compare with six SSL pretrained models: BYOL-A-v2 [11], SSAST [6], MAE-AST [7], Audio-MAE [9], BEATs [10] and M2D [15]. Sound event detection requires to perform frame-level multi-class classification.…”

“…ATST-Clip requires two different randomly cropped segments and the two segments have only a certain portion of overlap, while ATST-Frame asks for a frameto-frame correspondence between the two views. Besides, ATST-Clip and other clip-level contrastive audio pre-training methods [31] [5] [11] largely leverage the RRC augmentation [5] to achieve a good performance, but RRC will distort the frame-to-frame correspondence.…”

“…In terms of the training criterion, a portion of previous methods focus on learning global representation of an audio clip by using clip-level training criteria [3] [5] [11], while others propose learning local frame-wise or patch-wise representations by using frame-level [6] [7] or patch-level criteria [6] [7] [14] [15] [9] [10]. Most of the clip-level methods allow for extracting frame-wise representations from the intermediate output.…”

Digital Voice-Based Biomarker for Monitoring Respiratory Quality of Life: Findings from the Colive Voice Study

“…The eGeMAPS is a minimalistic set of acoustic Finally, we experiment with 4 different types of deep audio embeddings, i.e. VGGish [26], YAMNet, OpenL3 [27], and BYOL-A [28], which are state-of-the-art general audio features pretrained on large audio collections that are successfully used for a number of downstream tasks. Characteristics of different audio embeddings are provided in Table III.…”