Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval

Abstract

Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-based retrieval can work in practice -both in an absolute sense and versus alternative strategies that combine automatic speech recognition (ASR) with strong text encoders. In this work, we extensively study and expand choices of encoder architectures, training methodology (including unimodal and multimodal pretraining), and other factors. Our experiments cover different types of speech in three datasets: Flickr Audio, Places Audio, and Localized Narratives. Our best model configuration achieves large gains over state of the art, e.g., pushing recall-atone from 21.8% to 33.2% for Flickr Audio and 27.6% to 53.4% for Places Audio. We also show our best speech-based models can match or exceed cascaded ASR-to-text encoding when speech is spontaneous, accented, or otherwise hard to automatically transcribe.

Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset

“…The ResDAVEnet-VQ [27] architecture adds configurable vector quantization layers to the audio model. Several other models that learn audio-visual correspondences from both images and videos have been presented in recent work [28][29][30][31][32].…”

Cascaded Multilingual Audio-Visual Learning from Videos

“…Several works [2,3,15] demonstrate the ability to learn semantic relationships between objects in images and the spoken words describing them using only the pairing between images and spoken captions as supervision. Using this framework, researchers have proposed improved image encoders, audio encoders, and loss functions [4][5][6][7][8][16][17][18][19][20]. Harwath et al [3,4,21] collected 400k spoken audio captions of images in the Places205 [22] dataset in English, which is one of the largest spoken caption datasets.…”

Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset

“…The ResDAVEnet-VQ [27] architecture adds configurable vector quantization layers to the audio model. Several other models that learn audio-visual correspondences from both images and videos have been presented in recent work [28][29][30][31][32].…”