Pretext Tasks Selection for Multitask Self-Supervised Audio Representation Learning

Zaiem, Salah; Parcollet, Titouan; Essid, Slim; Heba, Abdelwahab

doi:10.1109/jstsp.2022.3195430

Cited by 9 publications

(4 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, the success of BERT in NLP has drawn attention from researchers in acoustic signal processing. Some pioneering works 7 – 10 , 21 , 22 have shown the effectiveness of adapting BERT and other self-supervised approaches to Automatic Speech Recognition (ASR). By designing pre-training objectives specific to the audio modality, it is possible to adapt BERT-like models to music and other audio domains.…”

Section: Related Workmentioning

confidence: 99%

Creating musical features using multi-faceted, multi-task encoders based on transformers

Greer

Shi

et al. 2023

Sci Rep

View full text Add to dashboard Cite

Computational machine intelligence approaches have enabled a variety of music-centric technologies in support of creating, sharing and interacting with music content. A strong performance on specific downstream application tasks, such as music genre detection and music emotion recognition, is paramount to ensuring broad capabilities for computational music understanding and Music Information Retrieval. Traditional approaches have relied on supervised learning to train models to support these music-related tasks. However, such approaches require copious annotated data and still may only provide insight into one view of music—namely, that related to the specific task at hand. We present a new model for generating audio-musical features that support music understanding, leveraging self-supervision and cross-domain learning. After pre-training using masked reconstruction of musical input features using self-attention bidirectional transformers, output representations are fine-tuned using several downstream music understanding tasks. Results show that the features generated by our multi-faceted, multi-task, music transformer model, which we call M3BERT, tend to outperform other audio and music embeddings on several diverse music-related tasks, indicating the potential of self-supervised and semi-supervised learning approaches toward a more generalized and robust computational approach to modeling music. Our work can offer a starting point for many music-related modeling tasks, with potential applications in learning deep representations and enabling robust technology applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Creating musical features using multi-faceted, multi-task encoders based on transformers

Greer

Shi

et al. 2023

Sci Rep

View full text Add to dashboard Cite

show abstract

“…This work extends this approach in two manners, first applying it for domain adaptation in a supervised setting, and second extending it to the speech recognition task. We use for this the Hilbert-Schmidt Independence Criterion (HSIC) [25], a kernel-based dependence estimator, also validated on pretext task selection in previous works [26]. The lower the HSIC estimator, the more conditionally independent the two sets are and the better the augmentations should be.…”

Section: Motivation and Technical Descriptionmentioning

confidence: 99%

“…Self-supervised learning (SSL) enables the use of large amounts of unlabelled data to obtain substantial performance improvements in a variety of downstream tasks without relying on manual annotations. Various approaches have been introduced including predictive coding [1,2], multi-task learning [3,4], autoencoding techniques [5] or contrastive learning [6,7]. In this context, data augmentation has become an important part of many self-supervised approaches.…”

Section: Introductionmentioning

confidence: 99%

Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learning

Zaiem¹,

Parcollet²,

Essid³

2022

Interspeech 2022

View full text Add to dashboard Cite

Self-Supervised Learning (SSL) has allowed leveraging large amounts of unlabeled speech data to improve the performance of speech recognition models even with small annotated datasets. Despite this, speech SSL representations may fail while facing an acoustic mismatch between the pretraining and target datasets. To address this issue, we propose a novel supervised domain adaptation method, designed for cases exhibiting such a mismatch in acoustic domains. It consists in applying properly calibrated data augmentations on a large clean dataset, bringing it closer to the target domain, and using it as part of an initial fine-tuning stage. Augmentations are automatically selected through the minimization of a conditional-dependence estimator, based on the target dataset. The approach is validated during an oracle experiment with controlled distortions and on two amateur-collected low-resource domains, reaching better performances compared to the baselines in both cases.

show abstract

“…This is particularly beneficial as it reduces the need for expensive and imprecise manual annotation. Various approaches have been introduced in the literature, including predictive coding, [1], multi-task learning [2,3], auto-encoding techniques [4] or contrastive learning [5].…”

Section: Introductionmentioning

confidence: 99%

Conditional Independence for Pretext Task Selection in Self-Supervised Speech Representation Learning

Zaiem¹,

Parcollet²,

Essid³

2021

Interspeech 2021

Self Cite

View full text Add to dashboard Cite

Through solving pretext tasks, self-supervised learning (SSL) leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. A common pretext task consists in pretraining a SSL model on pseudo-labels derived from the original signal. This technique is particularly relevant for speech data where various meaningful signal processing features may serve as pseudolabels. However, the process of selecting pseudo-labels, for speech or other types of data, remains mostly unexplored and currently relies on observing the results on the final downstream task. Nevertheless, this methodology is not sustainable at scale due to substantial computational (hence carbon) costs. Thus, this paper introduces a practical and theoretical framework to select relevant pseudo-labels with respect to a given downstream task. More precisely, we propose a functional estimator of the pseudo-label utility grounded in the conditional independence theory, which does not require any training. The experiments conducted on speaker recognition and automatic speech recognition validate our estimator, showing a significant correlation between the performance observed on the downstream task and the utility estimates obtained with our approach, facilitating the prospection of relevant pseudo-labels for selfsupervised speech representation learning.

show abstract

Pretext Tasks Selection for Multitask Self-Supervised Audio Representation Learning

Cited by 9 publications

References 66 publications

Creating musical features using multi-faceted, multi-task encoders based on transformers

Creating musical features using multi-faceted, multi-task encoders based on transformers

Automatic Data Augmentation Selection and Parametrization in Contrastive Self-Supervised Speech Representation Learning

Conditional Independence for Pretext Task Selection in Self-Supervised Speech Representation Learning

Contact Info

Product

Resources

About