Pretext Tasks selection for multitask self-supervised speech representation learning

Abstract

Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In various application domains, including computer vision, natural language processing and audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features has proven to be a particularly relevant pretext task leading to building useful self-supervised representations that prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks, where each task targets a different group of features for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates properly calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experiments conducted on speaker recognition and automatic speech recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.Preprint. Under review.

MV–MR: Multi-Views and Multi-Representations for Self-Supervised Learning and Knowledge Distillation

“…Such pretext tasks include, but are not limited to, applying and predicting parameters of the geometric transformations [ 26 ], jigsaw puzzle solving [ 27 ], inpainting [ 28 ] and colorization [ 29 ] of the images, and reversing augmentations. Typically, the pretext task methods have been coupled with other SSL techniques in recent years [ 30 , 31 , 32 ].…”

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

“…A well-defined pretext task should be defined in a way that enables the model to learn semantic features effectively from data. Nevertheless, it is generally challenging to define a task that can lead to meaningful a priori representations; it is difficult to establish if the surrogate task provides enough training signals to extract features that can be broadly usable by downstream tasks [158]. Therefore, many efforts have been made to design effective pretext tasks.…”

Sound and Visual Representation Learning with Multiple Pretraining Tasks

“…We call it Multi-SSL. While there have been a few works in this area on sound representation [60,72,69], language [66] or visual representation learning [16,22], they have only considered addressing this in the standard multi-tasking framework [16,67]. This work, however, introduces Multi-SSL which investigates different design options to combine multiple SSL tasks and provide insights into the downstream tasks.…”