Textual Supervision for Visually Grounded Spoken Language Understanding

Abstract

Visually-grounded models of spoken language understanding extract semantic information directly from speech, without relying on transcriptions. This is useful for low-resource languages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcriptions are available at training time. However, it is not clear how an end-to-end approach compares to a traditional pipeline-based approach when one has access to transcriptions. Comparing different strategies, we find that the pipeline approach works better when enough text is available. With low-resource languages in mind, we also show that translations can be effectively used in place of transcriptions but more data is needed to obtain similar results.

Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

“…Pasad, Shi, Kamper, and Livescu (2019) use a very similar approach while specifically focusing on low-resource settings and testing the effect of varying the amount of textual supervision. Higy, Elliott, and Chrupała (2020) investigate two forms of textual supervision in low-resource settings: transcriptions, and text translations. They also compare the multi-task learning approaches to simple pipeline architectures where text transcriptions are used to train an ASR module, and find that in most cases the pipeline is hard to improve on.…”

Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques

“…Pasad et al (2019) use a very similar approach while specifically focusing on low-resource settings and testing the effect of varying the amount of textual supervision. Higy et al (2020) investigate two forms of textual supervision in low-resource settings: transcriptions, and text translations. They also compare the multi-task learning approaches to simple pipeline architectures and find that in most cases pipeline is hard to improve on.…”

Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval

“…A number of works address cognitive and linguistic questions, such as understanding how different learned layers correspond to visual stimuli [3,4], learning linguistic units [5,6] or how visually grounded representations and data can help understand lexical competition in phonemic processing [7]. Other work addresses applied tasks, including multimodal retrieval [8,9,10], predicting written keywords given speech and image inputs [11], cross-modality alignment [12], retrieving speech in different languages using images as a pivot modality [13,14,15], and speech-to-speech retrieval [14,15]).…”

“…More recent work suggests that this same architecture can be used for a many tasks. However, only a few have specifically focus on improving retrieval [8,9], and no one has systematically evaluated the effectiveness of different design choices on multiple datasets.…”

“…Input speech representations: Prior work uses low-level features to represent audio: either Mel Frequency Cepstral Coefficient (MFCC) [8,9] or frequency band spectrograms [17,12]. Recently, self-supervised architectures trained with a contrastive loss showed improvements in English ASR [19] and multiple languages languages [20].…”