Improving Generalizability of Distilled Self-Supervised Speech Processing Models Under Distorted Settings

Huang, Kuo Chan; Yu-Kuan, Fu,; Hsu, Tsu-Yuan; Gutierrez, Fabian Ritter; Fan-Lin, Wang,; Tseng, Liang-Hsuan; Lee, Hung-yi

doi:10.1109/slt54892.2023.10022474

Cited by 6 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work has shown that data augmentation during the knowledge distillation process can improve the performance of compressed models in mismatched domain scenarios without compromising the model's size [18]. However, there are still limitations on how much robustness a model can acquire from the distillation process.…”

Section: Innovation #3: Environment Awarenessmentioning

confidence: 99%

“…While domain adaptation techniques, such as those proposed in "Robust HuBERT" [15] and "deHuBERT" [16], can alleviate this problem, the methods are not directly applicable for compression. Recent works have started to propose solutions that tackle compression and environmental robustness jointly (e.g., [17,18]). These solutions, however, have yet to be explored for ASR and have shown some sensitivity to varying environmental conditions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Environment-Aware Knowledge Distillation for Improved Resource-Constrained Edge Speech Recognition

Pimentel,

Guimarães,

Avila

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Recent advances in self-supervised learning have allowed automatic speech recognition (ASR) systems to achieve state-of-the-art (SOTA) word error rates (WER) while requiring only a fraction of the labeled data needed by its predecessors. Notwithstanding, while such models achieve SOTA results in matched train/test scenarios, their performance degrades substantially when tested in unseen conditions. To overcome this problem, strategies such as data augmentation and/or domain adaptation have been explored. Available models, however, are still too large to be considered for edge speech applications on resource-constrained devices; thus, model compression tools, such as knowledge distillation, are needed. In this paper, we propose three innovations on top of the existing DistilHuBERT distillation recipe: optimize the prediction heads, employ a targeted data augmentation method for different environmental scenarios, and employ a real-time environment estimator to choose between compressed models for inference. Experiments with the LibriSpeech dataset, corrupted with varying noise types and reverberation levels, show the proposed method outperforming several benchmark methods, both original and compressed, by as much as 48.4% and 89.2% in the word error reduction rate in extremely noisy and reverberant conditions, respectively, while reducing by 50% the number of parameters. Thus, the proposed method is well suited for resource-constrained edge speech recognition applications.

show abstract

Section: Innovation #3: Environment Awarenessmentioning

confidence: 99%