FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback

Goenka, Sonam; Zheng, Zhaoheng; Jaiswal, Ayush; Chada, Rakesh; Wu, Yue; Hedau, Varsha; Natarajan, Premkumar

doi:10.1109/cvpr52688.2022.01371

Cited by 55 publications

(16 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…TGIR evaluation. We compare our FAME-ViL with TGIR-specialist methods [2,22,48,75,90] and the art fashion-focused V+L model FashionViL [24] under the original protocol used by FashionIQ [83]. The results are given in Tab.…”

Section: Comparisons With Prior Art Methodsmentioning

confidence: 99%

“…(4) Fashion Image Captioning (FIC) generates a caption to describe the given image with semantically meaningful, fine-grained, and accurate words [85]. Many recent works have been trying to address these fashion tasks through VLP [21,22,24,33,55,87,94]. Most of them focus on the pre-training, then simply fine-tune the pre-trained model on each downstream task independently.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

Han¹,

Zhu²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

In the fashion domain, there exists a variety of visionand-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning. They differ drastically in each individual input/output format and dataset size. It has been common to design a task-specific model and fine-tune it independently from a pre-trained V+L model (e.g., CLIP). This results in parameter inefficiency and inability to exploit inter-task relatedness. To address such issues, we propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) in this work. Compared with existing approaches, FAME-ViL applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient. It is enabled by two novel components: (1) a task-versatile architecture with cross-attention adapters and task-specific adapters integrated into a unified V+L model, and (2) a stable and effective multi-task training strategy that supports learning from heterogeneous data and prevents negative transfer. Extensive experiments on four fashion tasks show that our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models. Code is available at https://github.com/BrandonHanx/FAME-ViL.

show abstract

Section: Comparisons With Prior Art Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

Han¹,

Zhu²,

Yu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the vision-language field, transformer architectures have achieved great success in many tasks, e.g., vision-and-language pre-training [38], [57], [58], image generation [59], visual question answering [60], open-vocabulary detection [61], image retrieval [62], vision-and-language navigation [63], etc. Lu et al [57] design a co-attention mechanism to incorporate languageattended vision features into language features.…”

Section: Transformermentioning

confidence: 99%

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Ding

Liu

Wang

et al. 2023

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

show abstract

“…More Applications. Besides VLP for standard VL tasks, VLP has also been applied to tackle (i) TextVQA (Singh et al, 2019) and TextCaps (Sidorov et al, 2020) tasks that require an AI system to comprehend scene text in order to perform VQA and captioning, such as TAP (Yang et al, 2021d) and LaTr (Biten et al, 2022); (ii) visual dialog (Das et al, 2017) that requires an AI system to chat about an input image, such as VisDial-BERT (Murahari et al, 2020) and VD-BERT (Wang et al, 2020b); (iii) fashion-domain tasks, such as Kaleido-BERT (Zhuge et al, 2021) and Fashion-VLP (Goenka et al, 2022); and (iv) vision-language navigation (VLN), such as PREVALENT (Hao et al, 2020) and VLN-BERT (Hong et al, 2021), to name a few. A detailed literature review on VLN can be found in Gu et al (2022b).…”

Section: Vlp For L Big Modelsmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback

Cited by 55 publications

References 38 publications

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Contact Info

Product

Resources

About