FLAVA: A Foundational Language And Vision Alignment Model

Singh, Amanpreet; Hu, Ronghang; Goswami, Vedanuj; Couairon, Guillaume; Galuba, Wojciech; Rohrbach, Marcus

doi:10.48550/arxiv.2112.04482

Cited by 21 publications

(28 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DeCLIP [26] utilized more image-text pairs collected from CLIP [30] by adding multiple self-supervised techniques. Inspired by BERT, other methods study cross-modal matching loss [8,28,34,35,44]. FLAVA [34] employs both contrastive and multimodal training objectives on paired and image-only datasets.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Masked Autoencoders Learn Transferable Representations

Geng¹,

Líu²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

Building scalable models to learn from diverse, multimodal data remains an open challenge. For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-specific encoders or contrastive learning, can learn transferable representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns a unified encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train on both paired image-text data as well as unpaired data.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Multimodal Masked Autoencoders Learn Transferable Representations

Geng¹,

Líu²,

Lee³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[7] and [56] propose a simple framework that can process information from multiple modalities with a uniform byte-sequence representation. [57] and [58] unify tasks of different modalities by designing various task-specific layers. [59] explores to employ a retrieval-based unified paradigm.…”

Section: Related Workmentioning

confidence: 99%

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Wang¹,

An²,

Men³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) to a simple sequence-tosequence learning framework based on the encoder-decoder architecture. OFA performs pretraining and finetuning with task instructions and introduces no extra task-specific layers for finetuning. Experimental results show that OFA achieves new state-of-the-arts on a series of multimodal tasks, including image captioning (COCO test CIDEr: 149.6), text-to-image generation (COCO test FID: 10.5), VQA (test-std acc.: 80.02), SNLI-VE (test acc.: 90.20), and referring expression comprehension (RefCOCO / RefCOCO+ / RefCOCOg test acc.: 92.93 / 90.10 / 85.20). Through extensive analyses, we demonstrate that OFA reaches comparable performance with uni-modal pretrained models (e.g., BERT, MAE, MoCo v3, SimCLR v2, etc.) in uni-modal tasks, including NLU, NLG, and image classification, and it effectively transfers to unseen tasks and domains. Code shall be released soon at https://github.com/OFA-Sys/OFA.

show abstract

“…SLIP [31] learned a joint representation by leveraging both a paired dataset, and a much larger image-only dataset, using self-supervised techniques. FLAVA [40] employs both contrastive and multi-modal training objectives on paired and image-only datasets. The joint representation was shown to hold a strong semantic alignment between the two modalities, enabling image generation [29,47], image manipulation [2,34], and image captioning [30].…”

Section: Retrieval For Generationmentioning

confidence: 99%

“…For Photo-Realistic experiments we use a modified version of the Public Multimodal Dataset (PMD) used by FLAVA [40]. The modified PMD dataset is composed from the following set of publicly available Text-Image datasets: SBU Captions [33], Localized Narratives [35], Conceptual Captions [38], Visual Genome [25], Wikipedia Image Text [42], Conceptual Captions 12M [7], Red Caps [9], and, a filtered version of YFCC100M [43].…”

Section: Datasetsmentioning

confidence: 99%

KNN-Diffusion: Image Generation via Large-Scale Retrieval

Ashual¹,

Sheynin²,

Polyak³

et al. 2022

Preprint

View full text Add to dashboard Cite

While the availability of massive Text-Image datasets is shown to be extremely useful in training large-scale generative models (e.g. DDPMs, Transformers), their output typically depends on the quality of both the input text, as well as the training dataset. In this work, we show how largescale retrieval methods, in particular efficient K-Nearest-Neighbors (KNN) search, can be used in order to train a model to adapt to new samples. Learning to adapt enables several new capabilities. Sifting through billions of records at inference time is extremely efficient and can alleviate the need to train or memorize an adequately large generative model. Additionally, fine-tuning trained models to new samples can be achieved by simply adding them to the table. Rare concepts, even without any presence in the training set, can be then leveraged during test time without any modification to the generative model. Our diffusion-based model trains on images only, by leveraging a joint Text-Image multi-modal metric. Compared to baseline methods, our generations achieve state of the art results both in human evaluations as well as with perceptual scores when tested on a public multimodal dataset of natural images, as well as on a collected dataset of 400 million Stickers.

show abstract

FLAVA: A Foundational Language And Vision Alignment Model

Cited by 21 publications

References 0 publications

Multimodal Masked Autoencoders Learn Transferable Representations

Multimodal Masked Autoencoders Learn Transferable Representations

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

KNN-Diffusion: Image Generation via Large-Scale Retrieval

Contact Info

Product

Resources

About