Debiased Video-Text Retrieval via Soft Positive Sample Calibration

Zhang, Huaiwen; Yang, Yang; Qi, Fan; Qian, Shengsheng; Xu, Changsheng

doi:10.1109/tcsvt.2023.3248873

Cited by 5 publications

(1 citation statement)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To enhance the cross-modal interaction capabilities of adapters, CALIP [21] leverages attention maps to fuse text and image features and inserts two fine-tunable linear layers before and after fusion. In addition, Cross-Modal Adapter (CMA) [32] and Multimodal Video Adapter (MV-Adapter) [94] achieve cross-modal interaction by sharing adapter weights between two modalities. These methods consider both single-modal and multi-modal scenarios but do not fully integrate the advantages of each modality.…”

Section: Multi-modal Adapter-based Fine-tuning Adaptationmentioning

confidence: 99%

Few-shot Adaptation of Multi-modal Foundation Models: A Survey

Liu,

Zhang,

Dai

et al. 2023

Preprint

View full text Add to dashboard Cite

Multi-modal models, such as CLIP, are replacing traditional supervised pre-training models (e.g., ImageNet-based pre-training) as the new generation of foundational visual models. These multi-modal models with robust and aligned semantic representations from billions of internet image-text pairs and can be applied to various downstream zero-shot tasks. However, in some fine-grained domains like medical imaging and remote sensing, the performance of multi-modal foundation models often leaves much to be desired. Consequently, many researchers have begun to explore few-shot adaptation methods for multi-modal foundation models, gradually deriving three technical approaches: 1) prompt-based fine-tuning adaptation methods, 2) adapter-based fine-tuning adaptation methods, and 3) adaptation methods based on external knowledge. Nevertheless, this rapidly developing field has produced numerous results without a comprehensive survey to systematically organize the research progress. Therefore, in this paper, we provide an extensive survey and analysis of the research advancements in few-shot adaptation methods for multi-modal models, summarizing commonly used datasets and experimental setups, and comparing the results of different methods. In addition, due to the lack of reliable theoretical support for existing methods, we derive the few-shot adaptation generalization error bound for multi-modal models. The theorem reveals that the generalization error of multi-modal foundation models is constrained by three factors: domain gap, model capacity, and sample size. Based on this, we propose three possible solutions from the following aspects: 1) domain distribution adaptation, 2) model selection adaptation, and 3) knowledge utilization adaptation.

show abstract

Section: Multi-modal Adapter-based Fine-tuning Adaptationmentioning

confidence: 99%