Conditional Independence for Pretext Task Selection in Self-Supervised Speech Representation Learning

Zaiem, Salah; Parcollet, Titouan; Essid, Slim

doi:10.21437/interspeech.2021-1027

Cited by 4 publications

(2 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These SSL embedding systems have been comprehensively benchmarked, considering factors such as model parameters and accuracy across various tasks [39]. For accessibility, pre-trained Automatic Speech Recognition (ASR) embedding models are readily available through the Hugging Face repository [40].…”

Section: Related Workmentioning

confidence: 99%

Analyzing wav2vec embedding in Parkinson’s disease speech: A study on cross-database classification and regression tasks

Klempir,

Krupicka

2024

Preprint

View full text Add to dashboard Cite

Advancements in deep learning speech representations have facilitated the effective use of extensive datasets comprised of unlabeled speech signals, and have achieved success in modeling tasks associated with Parkinson's disease (PD) with minimal annotated data. This study focuses on PD non-fine-tuned wav2vec 1.0 architecture. Utilizing features derived from wav2vec embedding, we develop machine learning models tailored for clinically relevant PD speech diagnosis tasks, such as cross-database classification and regression to predict demographic and articulation characteristics, for instance, modeling the subjects' age and number of characters per second. The primary aim is to conduct feature importance analysis on both classification and regression tasks, investigating whether latent discrete speech representations in PD are shared across models, particularly for related tasks. The proposed wav2vec-based models were evaluated on PD versus healthy controls using three multi-language-task PD datasets. Results indicated that wav2vec accurately detected PD based on speech, outperforming feature extraction using mel-frequency cepstral coefficients in the proposed cross-database scenarios. Furthermore, wav2vec proved effective in regression, modeling various quantitative speech characteristics related to intelligibility and aging. Subsequent analysis of important features, obtained using scikit-learn feature importance built-in tools and the Shapley additive explanations method, examined the presence of significant overlaps between classification and regression models. The feature importance experiments discovered shared features across trained models, with increased sharing for related tasks, further suggesting that wav2vec contributes to improved generalizability. In conclusion, the study proposes wav2vec embedding as a promising step toward a speech-based universal model to assist in the evaluation of PD.

show abstract

Section: Related Workmentioning

confidence: 99%

Analyzing wav2vec embedding in Parkinson’s disease speech: A study on cross-database classification and regression tasks

Klempir,

Krupicka

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The principal issue with conditional independence is the difficulty of computing good estimates of how much two variables are independent given a third one on realistic data [59]. In a previous work [73], we proposed a simple way to get an estimation of the conditional independence. This method has proven effective for individual pretext task selection, as the utility estimator correlates highly with the final downstream performances.…”

Section: Conditional Independence Estimator Computationmentioning

confidence: 99%

Pretext Tasks selection for multitask self-supervised speech representation learning

Zaiem¹,

Parcollet²,

Essid³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. In various application domains, including computer vision, natural language processing and audio/speech signal processing, a wide range of features where engineered through decades of research efforts. As it turns out, learning to predict such features has proven to be a particularly relevant pretext task leading to building useful self-supervised representations that prove to be effective for downstream tasks. However, methods and common practices for combining such pretext tasks, where each task targets a different group of features for better performance on the downstream task have not been explored and understood properly. In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks. This paper introduces a method to select a group of pretext tasks among a set of candidates. The method we propose estimates properly calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process. The experiments conducted on speaker recognition and automatic speech recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.Preprint. Under review.

show abstract

Pretext Tasks Selection for Multitask Self-Supervised Audio Representation Learning

Zaiem¹,

Parcollet

Essid³

et al. 2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

Conditional Independence for Pretext Task Selection in Self-Supervised Speech Representation Learning

Cited by 4 publications

References 59 publications

Analyzing wav2vec embedding in Parkinson’s disease speech: A study on cross-database classification and regression tasks

Analyzing wav2vec embedding in Parkinson’s disease speech: A study on cross-database classification and regression tasks

Pretext Tasks selection for multitask self-supervised speech representation learning

Pretext Tasks Selection for Multitask Self-Supervised Audio Representation Learning

Contact Info

Product

Resources

About