STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

Abstract

How to learn a better speech representation for end-to-end speech-to-text translation (ST) with limited labeled data? Existing techniques often attempt to transfer powerful machine translation (MT) capabilities to ST, but neglect the representation discrepancy across modalities. In this paper, we propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Specifically, we mix up the representation sequences of different modalities, and take both unimodal speech sequences and multimodal mixed sequences as input to the translation model in parallel, and regularize their output predictions with a selflearning framework. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy, and achieves significant improvements over a strong baseline on eight translation directions.

MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition

“…Speech recognition [42,6,21] is widely used in online meetings and social applications to recognize speech content. Speech translation [55,62,18] is commonly used in simultaneous interpretation applications for cross-lingual communication in crossborder travel and meetings. Keyword spotting [5,49,28] is employed in short video applications to quickly retrieve relevant content.…”

“…Many works [64,19,60,18,23] bridge the gap between modalities with mixup. [63] proposes mixup for data augmentation to improve model robustness.…”

“…Some [60,17] use CLIP [46] to retrieve semantically consistent images with text tokens and synthesize mixed sentences for text-visual consistency representation training. Others [56,18] construct manifold mixup interpolations based on semantic consistency between audio and text to enhance understanding of audio with textual datasets. By implementing the mixup strategy, these studies have shown notable improvements across a range of tasks, highlighting its potential to facilitate knowledge transfer between different modalities.…”

“…By implementing the mixup strategy, these studies have shown notable improvements across a range of tasks, highlighting its potential to facilitate knowledge transfer between different modalities. However, previous works use fixed hyperparameters [63] or mapping functions [18] for mixing ratios, which are typically not optimal and cannot be adapted to the training situation. In this paper, we propose an uncertainty-based [40] curriculum learning [7] strategy that gradually adjusts mixing ratios and apply mixup strategy for cross-modality knowledge transfer between audio and visual speeches for the first time.…”

“…As a tool for ultra-remote communication, many online interactions involve multiple languages, prompting the need for addressing cross-lingual challenges. Several works have attempted to tackle these challenges, including Machine Translation (MT) [9,34,14] for text utterance, Speech Translation (ST) [55,18] for audio utterance, and Speech-to-Speech Translation (S2ST) [55,18,16,31,27] for simultaneous interpretation. However, research on cross-lingual visual speech is still limited, as illustrated in Figure 1.…”