A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

Kishaan, Jeeveswaran,; Kathiresan, S.; Arnav, Varma,; Magdy, Omar; Zonooz, Bahram

doi:10.5220/0010917800003124

Cited by 5 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unimodal, task-specific neural networks have long been researched, with increasing convergence towards Transformer-based architectures [23,26] for both NLP [27] and Computer Vision [13,28,29].…”

Section: Related Workmentioning

confidence: 99%

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

Mustafa¹,

Riquelme²,

Puigcerver³

et al. 2022

Preprint

View full text Add to dashboard Cite

Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss. MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme. Across multiple scales, we demonstrate remarkable performance improvement over dense models of equivalent computational cost. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 84.1%, comparable to state-of-the-art methods which use larger custom per-modality backbones and pre-training schemes. We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts.

show abstract

Section: Related Workmentioning

confidence: 99%