Code-Switching without Switching: Language Agnostic End-to-End Speech Translation

Huber, Christian G.; Yavuz, Ugan, Enes; Waibel, Alexander

doi:10.48550/arxiv.2210.01512

Cited by 4 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Limitationsmentioning

confidence: 99%

Generating Synthetic Speech from SpokenVocab for Speech Translation

Zhao,

Haffari,

Shareghi

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Training end-to-end speech translation (ST) systems requires sufficiently large-scale data, which is unavailable for most language pairs and domains. One practical solution to the data scarcity issue is to convert text-based machine translation (MT) data to ST data via text-tospeech (TTS) systems.Yet, using TTS systems can be tedious and slow. In this work, we propose SpokenVocab, a simple, scalable and effective data augmentation technique to convert MT data to ST data on-the-fly. The idea is to retrieve and stitch audio snippets, corresponding to words in an MT sentence, from a spoken vocabulary bank. Our experiments on multiple language pairs show that stitched speech helps to improve translation quality by an average of 1.83 BLEU score, while performing equally well as TTS-generated speech in improving translation quality. We also showcase how Spo-kenVocab can be applied in code-switching ST for which often no TTS systems exit. 1

show abstract

Section: Limitationsmentioning

confidence: 99%

Generating Synthetic Speech from SpokenVocab for Speech Translation

Zhao,

Haffari,

Shareghi

2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

show abstract

“…Finally, for the end-to-end ST system, we used the language-agnostic model from Huber et al (2022) that can decode en-de ST and de ASR.…”

Section: Transcription and Translation Modelsmentioning

confidence: 99%

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation

Huber,

Dinh,

Mullov

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

View full text Add to dashboard Cite

The challenge of low-latency speech translation has recently draw significant interest in the research community as shown by several publications and shared tasks. Therefore, it is essential to evaluate these different approaches in realistic scenarios. However, currently only specific aspects of the systems are evaluated and often it is not possible to compare different approaches.In this work, we propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. The evaluation is carried out in an end-to-end fashion. This includes the segmentation of the audio as well as the run-time of the different components.Secondly, we compare different approaches to low-latency speech translation using this framework. We evaluate models with the option to revise the output as well as methods with fixed output. Furthermore, we directly compare stateof-the-art cascaded as well as end-to-end systems. Finally, the framework allows to automatically evaluate the translation quality as well as latency and also provides a web interface to show the low-latency model outputs to the user.

show abstract

“…CS ST exbihit difficulties (Huber et al, 2022;Weller et al, 2022), exposing several limitations in this study: 1) Bengali and Sanskrit (another minority language) are treated without difference, as they originate from the same script and Sanskrit is not supported by Google TTS. 2) We use a open-source language detection tool to calculate the oracle hyper-parameters in the dev set; yet, imperfection of the detector and the fact that source sentences are written in Latin regardless of the language deviate the scores from true values.…”

Section: Limitationsmentioning

confidence: 99%

Generating Synthetic Speech from SpokenVocab for Speech Translation

Zhao¹,

Haffar²,

Shareghi³

2022

Preprint

View full text Add to dashboard Cite

Training end-to-end speech translation (ST) systems requires sufficiently large-scale data, which is unavailable for most language pairs and domains. One practical solution to the data scarcity issue is to convert machine translation data (MT) to ST data via text-to-speech (TTS) systems. Yet, using TTS systems can be tedious and slow, as the conversion needs to be done for each MT dataset. In this work, we propose a simple, scalable and effective data augmentation technique, i.e., SpokenVocab, to convert MT data to ST data on-the-fly. The idea is to retrieve and stitch audio snippets from a SpokenVocab bank according to words in an MT sequence. Our experiments on multiple language pairs from Must-C show that this method outperforms strong baselines by an average of 1.83 BLEU scores, and it performs equally well as TTS-generated speech. We also showcase how SpokenVocab can be applied in code-switching ST for which often no TTS systems exit. 1 2 Only one work out of 8 uses TTS to augment data in the IWSLT2022 offline speech translation track.

show abstract

Code-Switching without Switching: Language Agnostic End-to-End Speech Translation

Cited by 4 publications

References 20 publications

Generating Synthetic Speech from SpokenVocab for Speech Translation

Generating Synthetic Speech from SpokenVocab for Speech Translation

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation

Generating Synthetic Speech from SpokenVocab for Speech Translation

Contact Info

Product

Resources

About