MURAL: Multimodal, Multitask Representations Across Languages

Abstract

Both image-caption pairs and translation pairs provide the means to learn deep representations of and connections between languages. We use both types of pairs in MURAL (MUltimodal, MUltitask Representations Across Languages), a dual encoder that solves two tasks: 1) image-text matching and 2) translation pair matching. By incorporating billions of translation pairs, MURAL extends ALIGN (Jia et al., 2021)-a state-of-the-art dual encoder learned from 1.8 billion noisy image-text pairs. When using the same encoders, MURAL's performance matches or exceeds ALIGN's cross-modal retrieval performance on wellresourced languages across several datasets. More importantly, it considerably improves performance on under-resourced languages, showing that text-text learning can overcome a paucity of image-caption examples for these languages. On the Wikipedia Image-Text dataset, for example, MURAL-BASE improves zero-shot mean recall by 8.1% on average for eight under-resourced languages and by 6.8% on average when fine-tuning. We additionally show that MURAL's text representations cluster not only with respect to genealogical connections but also based on areal linguistics, such as the Balkan Sprachbund.

SLIP: Self-supervision meets Language-Image Pre-training

“…Multi-modal multi-task learning. MURAL [30] extends ALIGN to the multi-lingual setting and introduces a crosslingual objective to improve multi-lingual image and text retrieval. Concurrently to this work, DeCLIP [40] adds several additional training objectives and more data collected in-house to CLIP in order to improve data efficiency.…”

Less is More: Generating Grounded Navigation Instructions from Landmarks

“…, r n ], we construct an m×n logit matrix A, where A i,j represents the compatibility of landmark phrase t i and frame r j . Logits are computed by combining signals from MURAL-large [31] -a highperforming multilingual, multimodal dual encoder trained on a mixture of 1.8b noisy image-text pairs and 6b translation pairs -and the RxR text timestamps, i.e. :…”

“…Including outbound landmarks gives a small boost in automatic evaluations. All landmarks are represented using 640-dimension image embeddings from MURAL-large [31]. Rewrite task.…”

“…4, the CenterNet landmark detector was trained using a single class to represent a landmark (i.e., 2 classes in total). In initial experiments, we also investigated training the detector using a richer set of landmark classes, by clustering landmark phrases (using kmeans) into classes based on their MURAL-large [31] text embeddings. However, this slightly reduced automatic evaluation scores for the full model (Tab.…”

“…Specifically, we bootstrap landmark annotations using text parsers to identify landmark phrases in instructions. We then use powerful image-text co-embedding models [31] combined with weak supervision from pose traces to ground those landmarks in the environment.…”