xGQA: Cross-Lingual Visual Question Answering

Abstract

Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset (Hudson and Manning, 2019) to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual, and-vice versa-multilingual models to become multimodal. Our proposed methods outperform current state-of-the-art multilingual multimodal models (e.g., M 3 P) in zeroshot cross-lingual settings, but the accuracy remains low across the board; a performance drop of around 38 accuracy points in target languages showcases the difficulty of zero-shot cross-lingual transfer for this task. Our results suggest that simple cross-lingual transfer of multimodal models yields latent multilingual multimodal misalignment, calling for more sophisticated methods for vision and multilingual language modeling. The xGQA dataset

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

“…Consequently, models trained on such datasets do not take into account linguistic diversity (Ponti et al, 2020) or cross-cultural nuances (Liu et al, 2021). 1 The need to expand V&L research towards more languages has been recognised by 1) the recent creation of multilingual training and evaluation data across diverse V&L tasks and languages (Srinivasan et al, 2021;Su et al, 2021;Pfeiffer et al, 2021;Liu et al, 2021;Wang et al, 2021, inter alia), as well as 2) the emergence of the first large multilingual-multimodal pretrained models (Ni et al, 2021;Zhou et al, 2021;Liu et al, 2021) and monolingual V&L models adapted to multiple languages (Chen et al, 2020;Pfeiffer et al, 2021). In this work, we merge and expand on these two research threads, aiming to highlight current achievements and challenges in this area and to facilitate comparative evaluations, thus bringing together the abovementioned collective research efforts.…”

“…We combine the text-only dataset SNLI (Bowman et al, 2015), with its multimodal (Xie et al, 2019) (Krishna et al, 2017). In particular, we use the English balanced training set to train our models, and evaluate on the few-shot evaluation sets defined by Pfeiffer et al (2021) to allow for direct comparison between zero-shot and few-shot experiments.…”

“…The lack of a standard set for few-shot learning harms the replicability and comparability of such experiments, given that model performance exhibits a strong sensitivity to the selection of few-shot examples (Zhao et al, 2021). Therefore, following Schick & Schütze (2021) and Pfeiffer et al (2021), in addition to the train / validation / test splits, we also release standard few-shot splits for every task and language (see Appendix A for details on how each split is defined).…”

“…Few-shot Data. We reuse the few-shot splits defined by Pfeiffer et al (2021), which also include development data in the target languages. The authors sampled 48 images, giving up to 1,490 questions translated for each language.…”

“…Nonetheless, this trend is reversing by virtue of a series of independent efforts in the the V&L research community: on top of expanding multilingual image-sentence retrieval to lower-resource languages (Srinivasan et al, 2021), we have recently witnessed pioneering work on multimodal machine translation (Yao & Wan, 2020;Huang et al, 2020), multilingual visual question answering (Pfeiffer et al, 2021), multilingual text-to-video search (Huang et al, 2021;Lei et al, 2021), multilingual visual reasoning (Liu et al, 2021), and cross-modal retrieval (Jain et al, 2021). Yet, multilingual multimodal models (Ni et al, 2021;Jain et al, 2021) are typically tested only on the task of cross-modal retrieval.…”