Adapting multilingual speech representation model for a new, underresourced language through multilingual fine-tuning and continued pretraining

Nowakowski, Karol; Ptaszyński, Michał; Murasaki, Kyoko; Nieuważny, Jagna

doi:10.1016/j.ipm.2022.103148

Cited by 17 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It may be that CPT only yields appreciable performance gains once a sufficient amount of unlabeled audio can be obtained (e.g. 200 hours of Ainu: Nowakowski et al, 2023). However, obtaining such a large amount of data for minority languages or language variants such as Gronings, Besemah, and Nasal is unlikely.…”

Section: Discussionmentioning

confidence: 99%

Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Bartelds¹,

San²,

McDonnell³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

The performance of automatic speech recognition (ASR) systems has advanced substantially in recent years, particularly for languages for which a large amount of transcribed speech is available. Unfortunately, for low-resource languages, such as minority languages, regional languages or dialects, ASR performance generally remains much lower. In this study, we investigate whether data augmentation techniques could help improve low-resource ASR performance, focusing on four typologically diverse minority languages or language variants (West Germanic: Gronings, West-Frisian; Malayo-Polynesian: Besemah, Nasal). For all four languages, we examine the use of selftraining, where an ASR system trained with the available human-transcribed data is used to generate transcriptions, which are then combined with the original data to train a new ASR system. For Gronings, for which there was a preexisting text-to-speech (TTS) system available, we also examined the use of TTS to generate ASR training data from text-only sources. We find that using a self-training approach consistently yields improved performance (a relative WER reduction up to 20.5% compared to using an ASR system trained on 24 minutes of manually transcribed speech). The performance gain from TTS augmentation for Gronings was even stronger (up to 25.5% relative reduction in WER compared to a system based on 24 minutes of manually transcribed speech). In sum, our results show the benefit of using selftraining or (if possible) TTS-generated data as an efficient solution to overcome the limitations of data availability for resource-scarce languages in order to improve ASR performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Bartelds¹,

San²,

McDonnell³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Subsequent advancements by Nowakowski [Nowakowski et al, 2023] involved refining a multilingual speech representation model for lesser-resourced languages via multilingual fine-tuning and ongoing pretraining, illustrating the adaptability of NLP techniques to languages with limited resources like Ainu.…”

Section: Nlp For Ainumentioning

confidence: 99%

Ainu–Japanese Bi-directional Neural Machine Translation: A Step Towards Linguistic Preservation of Ainu, An Under-Resourced Indigenous Language in Japan

Miyagawa

2024

Journal of Data Mining &Amp; Digital Humanities

View full text Add to dashboard Cite

This study presents a groundbreaking approach to preserving the Ainu language, recognized as critically endangered by UNESCO, by developing a bi-directional neural machine translation (MT) system between Ainu and Japanese. Utilizing the Marian MT framework, known for its effectiveness with resource-scarce languages, the research aims to overcome the linguistic complexities inherent in Ainu's polysynthetic structure. The paper delineates a comprehensive methodology encompassing data collection from diverse Ainu text sources, meticulous preprocessing, and the deployment of neural MT models, culminating in the achievement of significant SacreBLEU scores that underscore the models' translation accuracy. The findings illustrate the potential of advanced MT technology to facilitate linguistic preservation and educational endeavors, advocating for integrating such technologies in safeguarding endangered languages. This research not only underscores the critical role of MT in bridging language divides but also sets a precedent for employing computational linguistics to preserve cultural and linguistic heritage.

show abstract

“…Cross-dataset evaluation for popular English speech corpora indicates that CPT helps to reduce the error rate in the target domain. In [43] and [11] CPT is utilized for cross-lingual adaptation of wav2vec2 for Korean and Ainu respectively. Notably for Ainu, which is an endagered language, CPT has resulted in significant system In the right, we see the proposed domain-adaptive finetuning stage, where the speech recognition task is learned using transcribed source domain data, while adaptation to the target domain is performed by including the self-supervised loss over (audio-only) source and target domain data improvement.…”

Section: Leveraging In-domain Self-supervisionmentioning

confidence: 99%

“…Furthermore, UDA has been used for speaker adaptation, and to improve performance under speaker, gender and accent variability [7], [8]. UDA has also been employed for multilingual and cross-lingual ASR, in order to improve ASR models for low-resource languages [9], adapt to different dialects [10], and even train speech recognition systems for endangered languages [11].…”

Section: Introductionmentioning

confidence: 99%

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Paraskevopoulos¹,

Kouzelis²,

Rouvalis³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Modern speech recognition systems exhibits rapid performance degradation under domain shift. This issue is especially prevalent in data-scarce settings, such as low-resource languages, where diversity of training data is limited.</p> <p>In this work we propose M2DS2, a simple and sample-efficient finetuning strategy for large pretrained speech models, based on mixed source and target domain self-supervision. We find that including source domain self-supervision stabilizes training and avoids mode collapse of the latent representations. For evaluation, we collect HParl, a 120 hour speech corpus for Greek, consisting of plenary sessions in the Greek Parliament. We merge HParl with two popular Greek corpora to create GREC-MD, a test-bed for multi-domain evaluation of Greek ASR systems. In our experiments we find that, while other Unsupervised Domain Adaptation baselines fail in this resource-constrained environment, M2DS2 yields significant improvements for cross-domain adaptation, even when a only a few hours of in-domain audio are available. When we relax the problem in a weakly supervised setting, we find that independent adaptation for audio using M2DS2 and language using simple LM augmentation techniques is particularly effective, yielding word error rates comparable to the fully supervised baselines.</p>

show abstract

Adapting multilingual speech representation model for a new, underresourced language through multilingual fine-tuning and continued pretraining

Cited by 17 publications

References 19 publications

Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Ainu–Japanese Bi-directional Neural Machine Translation: A Step Towards Linguistic Preservation of Ainu, An Under-Resourced Indigenous Language in Japan

Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems: A case study for Modern Greek

Contact Info

Product

Resources

About