Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation

Huang, Kuo-Liang; Yu-Kuan, Fu,; Lee, Hung-yi

doi:10.21437/interspeech.2022-519

Cited by 15 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the failure modes of SSL models are still poorly understood, and it remains unclear whether they provide more or less robustness to adversarial attacks than fully supervised models. Due to the importance of this research direction, while writing this paper, there is already some related research about enhancing the robustness of SSL models [281], [352]- [354] and identifying their vulnerability to adversarial attack [351].…”

Section: Discussionmentioning

confidence: 99%

Self-Supervised Speech Representation Learning: A Review

Mohamed

Lee

Borgholt

et al. 2022

IEEE J. Sel. Top. Signal Process.

169

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. Users may download and print one copy of any publication from the public portal for the purpose of private study or research.  You may not further distribute the material or use it for any profit-making activity or commercial gain  You may freely distribute the URL identifying the publication in the public portal If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

show abstract

Section: Discussionmentioning

confidence: 99%

Self-Supervised Speech Representation Learning: A Review

Mohamed

Lee

Borgholt

et al. 2022

IEEE J. Sel. Top. Signal Process.

169

View full text Add to dashboard Cite

show abstract

“…While larger versions of HuBERT have shown improved ASR accuracy in noisy conditions [17], for edge applications such as large footprint models, they can be problematic, and smaller models can be more sensitive to noise. To this end, the authors in [15] proposed the so-called Robust HuBERT method, where domain adversarial training was used to make the system more robust to environmental factors. More specifically, a domain discriminator is responsible for classifying the source of the distortions applied to the utterance.…”

Section: Making Hubert Robust To Noisementioning

confidence: 99%

“…Unseen noise and reverberation levels, for example, are known to drastically reduce the accuracy of even state-of-the-art ASR systems [12] and can be highly sensitive to different environmental conditions [13,14]. While domain adaptation techniques, such as those proposed in "Robust HuBERT" [15] and "deHuBERT" [16], can alleviate this problem, the methods are not directly applicable for compression. Recent works have started to propose solutions that tackle compression and environmental robustness jointly (e.g., [17,18]).…”

Section: Introductionmentioning

confidence: 99%

Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition

Pimentel,

Guimarães,

Avila

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Recent advances in self-supervised learning have allowed automatic speech recognition (ASR) systems to achieve state-of-the-art (SOTA) word error rates (WER) while requiring only a fraction of the labeled data needed by its predecessors. Notwithstanding, while such models achieve SOTA results in matched train/test scenarios, their performance degrades substantially when tested in unseen conditions. To overcome this problem, strategies such as data augmentation and/or domain adaptation have been explored. Available models, however, are still too large to be considered for edge speech applications on resource-constrained devices; thus, model compression tools, such as knowledge distillation, are needed. In this paper, we propose three innovations on top of the existing DistilHuBERT distillation recipe: optimize the prediction heads, employ a targeted data augmentation method for different environmental scenarios, and employ a real-time environment estimator to choose between compressed models for inference. Experiments with the LibriSpeech dataset, corrupted with varying noise types and reverberation levels, show the proposed method outperforming several benchmark methods, both original and compressed, by as much as 48.4% and 89.2% in the word error reduction rate in extremely noisy and reverberant conditions, respectively, while reducing by 50% the number of parameters. Thus, the proposed method is well suited for resource-constrained edge speech recognition applications.

show abstract

“…SPIN employs speaker invariant clustering to improve content representations. The term SSFT was proposed in [4] to distinguish fine-tuning methods using only audio [5,6] from supervised fine-tuning using labelled data [7].…”

Section: Introductionmentioning

confidence: 99%

SCORE: Self-Supervised Correspondence Fine-Tuning for Improved Content Representations

Meghanani,

Hain

2024

ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

There is a growing interest in cost-effective self-supervised fine-tuning (SSFT) of self-supervised learning (SSL)-based speech models to obtain task-specific representations. These task-specific representations are used for robust performance on various downstream tasks by fine-tuning on the labelled data. This work presents a cost-effective SSFT method named Self-supervised Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for content-related tasks. The proposed method uses a correspondence training strategy, aiming to learn similar representations from perturbed speech and original speech. Commonly used data augmentation techniques for content-related tasks (ASR) are applied to obtain perturbed speech. SCORE fine-tuned Hu-BERT outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of fine-tuning (< 5 hrs) on a single GPU for automatic speech recognition, phoneme recognition, and query-by-example tasks, with relative improvements of 1.09%, 3.58%, and 12.65%, respectively. SCORE provides competitive results with the recently proposed SSFT method SPIN, using only 1/3 of the processed speech compared to SPIN.

show abstract

Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation

Cited by 15 publications

References 0 publications

Self-Supervised Speech Representation Learning: A Review

Self-Supervised Speech Representation Learning: A Review

Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition

SCORE: Self-Supervised Correspondence Fine-Tuning for Improved Content Representations

Contact Info

Product

Resources

About