Clover: Towards A Unified Video-Language Alignment and Fusion Model

Huang, Jingjia; Li, Yinan; Feng, Jiashi; Sun, Xingming; Ji, Rongrong

doi:10.48550/arxiv.2207.07885

Cited by 4 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yiwei et al [24] presented the X-CLIP model, a multi-grain contrastive model for video-text retrieval. Jingjia et al [25] proposed the Clover approach to pretraining linked video language, which sets a new state-of-the-art on several downstream tasks. Haofan et al [26] suggested a video-text retrieval strategy for cross-modal representation learning, which aims to select the video from a pool of candidate videos that matches the text query.…”

Section: Text Regain Methodsmentioning

confidence: 99%

Augmented Reality and Ontology-based Approach for Enhancing Arabic Text Recognition and Visualization

Ouali

Halima

MASMOUDI

et al. 2023

Preprint

View full text Add to dashboard Cite

Written text is a commonly used means of transferring information in our daily lives, and it is especially crucial in unfamiliar environments where we rely on written text to achieve our goals. In such scenarios, signs or signboards help us navigate and determine the direction we should take. However, various factors, such as prolonged exposure to the text, text obstructions, and poor lighting, may hinder our ability to read the text accurately, leading to erroneous readings. A single incorrectly read letter can lead to a false interpretation of the entire text, potentially resulting in serious consequences. To address this issue, we propose an approach to correct letter imperfections in a word and display it in an improved manner. Our approach consists of two phases: the pre-treatment phase and the letter imperfection recovery phase. The pre-treatment phase relies on Augmented Reality technology to detect and recognize text with letter imperfections. In the letter imperfection recovery phase, we employ ontology and a database to correct the letter imperfections. To enhance the accuracy of our results, we apply Augmented Reality technology to obtain a 3D image for perfect visualization. We conducted an evaluation and experimentation phase by administering a questionnaire to a group of participants to obtain feedback on our approach. The results were encouraging and demonstrated the effectiveness of our approach.

show abstract

Section: Text Regain Methodsmentioning

confidence: 99%

Augmented Reality and Ontology-based Approach for Enhancing Arabic Text Recognition and Visualization

Ouali

Halima

MASMOUDI

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, it is hard to get a pretrained model as powerful as CLIP in the video domain due to the unaffordable demands on computation resources and the difficulty of collecting video-text data pairs as large and diverse as image-text data. Instead of directly pursuing video-text pretrained models [17,27], a potential alternative solution that benefits video downstream tasks is to transfer the knowledge in image-text pretrained models to the video domain, which has attracted increasing attention in recent years [12,13,26,29,30,41].…”

Section: Introductionmentioning

confidence: 99%

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring

Liu¹,

Huang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Image-text pretrained models, e.g., CLIP, have shown impressive general multi-modal knowledge learned from large-scale image-text data pairs, thus attracting increasing attention for their potential to improve visual representation learning in the video domain. In this paper, based on the CLIP model, we revisit temporal modeling in the context of image-to-video knowledge transferring, which is the key point for extending image-text pretrained models to the video domain. We find that current temporal modeling mechanisms are tailored to either high-level semanticdominant tasks (e.g., retrieval) or low-level visual patterndominant tasks (e.g., recognition), and fail to work on the two cases simultaneously. The key difficulty lies in modeling temporal dependency while taking advantage of both highlevel and low-level knowledge in CLIP model. To tackle this problem, we present Spatial-Temporal Auxiliary Network (STAN) -a simple and effective temporal modeling mechanism extending CLIP model to diverse video tasks. Specifically, to realize both low-level and high-level knowledge transferring, STAN adopts a branch structure with decomposed spatial-temporal modules that enable multilevel CLIP features to be spatial-temporally contextualized. We evaluate our method on two representative video tasks: Video-Text Retrieval and Video Recognition. Extensive experiments demonstrate the superiority of our model over the state-of-the-art methods on various datasets, including MSR-VTT, DiDeMo, LSMDC, MSVD, Kinetics-400, and Something-Something-V2. Codes will be available at https://github.com/farewellthree/STAN

show abstract

“…Text-based video retrieval tasks, including text-based video retrieval (Luo et al 2020;Zhu and Yang 2020;Li et al 2020) and text-based video corpus moment retrieval (Li et al 2020) have shown significant potential and alluring technological value. Thanks to the ability of cross-modality alignments, video-language pre-training shows effectiveness on these retrieval tasks (Wang et al 2022b;Huang et al 2022). Cross-modal alignment is the key challenge in learning video-language pre-training.…”

Section: Introductionmentioning

confidence: 99%

Learning Semantic Alignment with Global Modality Reconstruction for Video-Language Pre-training towards Retrieval

Shi

Leng

et al. 2023

AAAI

View full text Add to dashboard Cite

Video-language pre-training for text-based video retrieval tasks is vitally important. Previous pre-training methods suffer from the semantic misalignments. The reason is that these methods ignore sequence alignments but focusing on critical token alignment. To alleviate the problem, we propose a video-language pre-training framework, termed videolanguage pre-training For lEarning sEmantic aLignments (FEEL), to learn semantic alignments at the sequence level. Specifically, the global modality reconstruction and the cross- modal self-contrasting method is utilized to learn the alignments at the sequence level better. Extensive experimental results demonstrate the effectiveness of FEEL on text-based video retrieval and text-based video corpus moment retrieval.

show abstract

Clover: Towards A Unified Video-Language Alignment and Fusion Model

Cited by 4 publications

References 37 publications

Augmented Reality and Ontology-based Approach for Enhancing Arabic Text Recognition and Visualization

Augmented Reality and Ontology-based Approach for Enhancing Arabic Text Recognition and Visualization

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring

Learning Semantic Alignment with Global Modality Reconstruction for Video-Language Pre-training towards Retrieval

Contact Info

Product

Resources

About