Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech

Tomanek, Katrin; Zayats, Vicky; Padfield, Dirk; Vaillancourt, Kara; Biadsy, Fadi

doi:10.48550/arxiv.2109.06952

Cited by 1 publication

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We find that fine-tuning the entire model (i.e., adapting all parameters of the Basemodel including encoder and decoders) for each speaker exerts substantial improvements across all our speakers with an average WER of 14.2%. This is consistent with our previous work that Parrotron model fine-tuning achieves high quality model personalization for atypical speech [4,23,18,12]. For this experiment, we employ the same model architecture and the same fine-tuning procedure described in [18].…”

Section: Basemodel Vs Model Fine-tuning Resultssupporting

confidence: 76%

“…We choose residual adapter layers [10] as our choice of Submodels for several reasons: (1) Adapter layers can easily be added to the encoder which is generally responsible for modeling the acoustic-phonetic of speech from the acoustic; signal, (2) Due to their residual connection one can disable the Submodel by simply setting the residual factor to zero, reverting the model to the Basemodel; (3) The size of this Submodel can be easily controlled by a bottleneck dimension; (4) Controlling the bottleneck dimension is internal to the Submodel, allowing the use of a pre-compiled and optimized execution graph for fast inference while being able to replace the tensors shape dynamically; (5) We have seen previously that adapters successfully model atypical and accented speech for ASR personalization and specialization. [12]…”

Section: Submodelsmentioning

confidence: 99%

“…Our Submodel consists of 17 residual adapter blocks, each added between each layer of the 17-layer conformer encoder in the Parrotron Basemodel. [10,11,12,25] Each adapter block starts with layer normalization, followed by a downprojection to a bottleneck dimension d b followed by Relu, and finally another feed-forward layer with up-projection to the original input dimension, as shown in Figure 1. The bottleneck dimension d b enables us to control the number of parameters of each adapter.…”

Section: Parrotron Submodelmentioning

confidence: 99%

“…Light-weight adaptation techniques, such as [7,8,9,10,11,12,13,14] have shown significant reduction on memory, computation and storage costs by learning or adapting significantly less parameters than that of the full model. We are inspired by this work to propose a framework that scales both training and serving for many use-cases.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization

Biadsy¹,

Youzheng²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Model fine-tuning and adaptation have become a common approach for model specialization for downstream tasks or domains. Fine-tuning the entire model or a subset of the parameters using light-weight adaptation has shown considerable success across different specialization tasks. Fine-tuning a model for a large number of domains typically requires starting a new training job for every domain posing scaling limitations. Once these models are trained, deploying them also poses significant scalability challenges for inference for real-time applications. In this paper, building upon prior light-weight adaptation techniques, we propose a modular framework that enables us to substantially improve scalability for model training and inference. We introduce Submodels that can be quickly and dynamically loaded for on-the-fly inference. We also propose multiple approaches for training those Submodels in parallel using an embedding space in the same training job. We test our framework on an extreme use-case which is speech model personalization for atypical speech, requiring a Submodel for each user. We obtain 128× Submodel throughput with a fixed computation budget without a loss of accuracy. We also show that learning a speaker-embedding space can scale further and reduce the amount of personalization training data required per speaker.

show abstract