CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Luo, Huaishao; Ji, Lei; Zhong, Ming; Chen, Yang; Wen, Lijie; Duan, Nan; Li, Tianrui

doi:10.1016/j.neucom.2022.07.028

Cited by 394 publications

(179 citation statements)

References 32 publications

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…We report the precision@N , recall@N and mean rank (MR) following existing work on video retrieval (Luo et al, 2021) (see §B.2 for metric definitions). Table 3 lists the results.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data

Zhou¹,

Zhang²,

Yang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Procedures are inherently hierarchical. To make videos, one may need to purchase a camera, which in turn may require one to set a budget. While such hierarchical knowledge is critical for reasoning about complex procedures, most existing work has treated procedures as shallow structures without modeling the parent-child relation. In this work, we attempt to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow, a website containing more than 110k instructional articles, each documenting the steps to carry out a complex procedure. To this end, we develop a simple and efficient method that links steps (e.g., purchase a camera) in an article to other articles with similar goals (e.g., how to choose a camera), recursively constructing the KB. Our method significantly outperforms several strong baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval. 1 * Equal contribution. 1 A demo with partial data can be found at https://wikihow-hierarchy.github.io/. The code and the data are at https://github.com/shuyanzhou/wikihow_hierarchy.

show abstract

“…We report the precision@N , recall@N and mean rank (MR) following existing work on video retrieval (Luo et al, 2021) (see §B.2 for metric definitions). Table 3 lists the results.…”

Section: Resultsmentioning

confidence: 99%

“…We report precision@N , recall@N and mean rank (MR) following existing works on video retrieval (Luo et al, 2021)…”

Section: B2 Evaluation Metricsmentioning

confidence: 99%

Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data

Zhou¹,

Zhang²,

Yang³

et al. 2022

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…CLIP [57] was an early example of achieving impressive zeroshot visual classification following self-supervised training with over 400M image-caption pairs. Similar advances have been made for audio [29] and video [50] by utilizing different augmented views of the same modality [19,18,27,60], or by learning joint embeddings of short [29,3,50] or longform [39] videos. Our system leverages such work for learning video representations that capture matching video pairs for the task of match cutting.…”

Section: Related Workmentioning

confidence: 97%

Match Cutting: Finding Cuts with Smooth Visual Transitions

Chen¹,

Ziai²,

Tucker³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…CLIP4Clip [12] is a model that achieved SOTA results on several video-text retrieval datasets. It is not particularly innovative from the model with pretrained CLIP model, through transfer learning, or finetune to complete the task of video retrieval.…”

Section: Retrieval Tasksmentioning

confidence: 99%

A Survey on CLIP-Guided Vision-Language Tasks

Yu¹

2022

HSET

View full text Add to dashboard Cite

Multimodal learning refers to the representation of different modalities using a unified model. Modalities include images, text, audio, etc. In this article, we will first introduce the basic approach of CLIP which is a vision language model with the power of connecting different modalities, and then present different models inspired by CLIP on various downstream tasks. In the end, we conclude with a summary of the prospects and limitations of multimodal learning.

show abstract

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Cited by 394 publications

References 32 publications

Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data

Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data

Match Cutting: Finding Cuts with Smooth Visual Transitions

A Survey on CLIP-Guided Vision-Language Tasks

Contact Info

Product

Resources

About