Multimodal Masked Autoencoders Learn Transferable Representations

Abstract

Building scalable models to learn from diverse, multimodal data remains an open challenge. For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-specific encoders or contrastive learning, can learn transferable representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns a unified encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train on both paired image-text data as well as unpaired data.Preprint. Under review.

Masked World Models for Visual Control

“…It would also be interesting to study introducing auxiliary prediction for other modalities, such as audio. Another weakness is that our model operates only on RGB pixels from a single camera viewpoint; we look forward to a future work that incorporates different input modalities such as proprioceptive states and point clouds, building on top of the recent multi-modal learning approaches [52,53]. Finally, our approach trains behaviors from scratch, which makes it still too sample-inefficient to be used in real-world scenarios.…”

Learning Customized Visual Models with Retrieval-Augmented Knowledge

“…The pioneering works of CLIP [57] and ALIGN [31] make use of contrastive learning to pretrain models on billion-scale web-crawled image-text pairs. There are an increasing number of studies to improve their generality from various modeling perspectives, including training objectives [17,19,20,52,82,85], scaling techniques [15,54,82], data efficiency [38,41], and leveraging multilingual correlations [15,30]. In academia, several works demonstrate techniques to improve the learned semantic representations on datasets at a smaller scale (e.g.…”

Self Supervision Does Not Help Natural Language Supervision at Scale

“…Masking across modalities: BART [51] applies a similar strategy to text data, but uses a cross-entropy loss over the masked-token output distribution and a far lower masking ratio (typically around 15%). M3AE [31] extends MAE and BART to incorporate inputs from both text and image modalities, using per-modality input and output projections. Otherwise the encoder-decoder architecture resembles MAE, with the addition of learned modality-indicating embeddings to each transformer input.…”