Prenatal Genetic Analysis of Kidney Abnormalities

Shen, Yongmei; Li, Yaqi; Zhao, Xin; Li, Wen; Zhang, Lei; Yao, Lei; Cao, Jiasong; Li, Shanshan; Zhuo, Wei; Chang, Ying

doi:10.21203/rs.3.rs-2575309/v1

Cited by 1 publication

(1 citation statement)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ability to understand and generate natural language from visual information is a critical component of many real-world applications, including visual question answering (VQA), visual reasoning, and multimodal information retrieval. In recent years, the success of deep learning in natural language processing (NLP) has led to the development of large-scale vision-language models (VLMs) (Tan and Bansal, 2019;Li et al, 2021b;Kim et al, 2021a;Alayrac et al, 2022;Wang et al, 2022c;Shen et al, 2022b;Li et al, 2021a;Shen et al, 2022a;Jia et al, 2021; that leverage powerful neural network architectures to encode and decode multimodal information. However, state-of-the-art vision-language models like Flamingo-80B (Alayrac et al, 2022), BEIT-3-1.9B , and PaLI-17B can be computationally expensive and difficult to train, which has motivated researchers to explore ways of improving their efficiency and effectiveness.…”

Section: Introductionmentioning

confidence: 99%

Scaling Vision-Language Models with Sparse Mixture of Experts

Shen,

Yao,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These models aim to bridge the gap between text and visual information, enabling a more comprehensive understanding of multimedia data. However, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to addressing this challenge is the use of sparsely-gated mixture-of-experts (MoE) techniques, which divide the model into smaller, specialized submodels that can jointly solve a task. In this paper, we explore the effectiveness of MoE in scaling vision-language models, demonstrating its potential to achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale vision-language models and other multimodal machine learning applications. * equal contribution; § work initiated during an internship at Microsoft. Code is available at https://vlmoe. github.io.

show abstract