Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer

Jiang, Hao; Mu, Yadong

doi:10.1109/cvpr52688.2022.01590

Cited by 28 publications

(4 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…on identifying temporal moments in videos based on natural language queries, has garnered escalating interest from both the academic and industrial sectors. Video moment localization holds a broad spectrum of applications, including video highlight detection (Liu et al 2022), video summarization (Jiang and Mu 2022), etc.…”

Section: Previous Methodsmentioning

confidence: 99%

Transferable Video Moment Localization by Moment-Guided Query Prompting

Jiang,

Yizhang,

2024

AAAI

View full text Add to dashboard Cite

Video moment localization stands as a crucial task within the realm of computer vision, entailing the identification of temporal moments in untrimmed videos that bear semantic relevance to the supplied natural language queries. This work delves into a relatively unexplored facet of the task: the transferability of video moment localization models. This concern is addressed by evaluating moment localization models within a cross-domain transfer setting. In this setup, we curate multiple datasets distinguished by substantial domain gaps. The model undergoes training on one of these datasets, while validation and testing are executed using the remaining datasets. To confront the challenges inherent in this scenario, we draw inspiration from the recently introduced large-scale pre-trained vision-language models. Our focus is on exploring how the strategic utilization of these resources can bolster the capabilities of a model designed for video moment localization. Nevertheless, the distribution of language queries in video moment localization usually diverges from the text used by pre-trained models, exhibiting distinctions in aspects such as length, content, expression, and more. To mitigate the gap, this work proposes a Moment-Guided Query Prompting (MGQP) method for video moment localization. Our key idea is to generate multiple distinct and complementary prompt primitives through stratification of the original queries. Our approach is comprised of a prompt primitive constructor, a multimodal prompt refiner, and a holistic prompt incorporator. We carry out extensive experiments on Charades-STA, TACoS, DiDeMo, and YouCookII datasets, and investigate the efficacy of the proposed method using various pre-trained models, such as CLIP, ActionCLIP, CLIP4Clip, and VideoCLIP. The experimental results demonstrate the effectiveness of our proposed method.

show abstract

Section: Previous Methodsmentioning

confidence: 99%

Transferable Video Moment Localization by Moment-Guided Query Prompting

Jiang,

Yizhang,

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Evaluation protocol. Following existing works [13,26,27,37,55], we evaluate the proposed method under the same setting. TVSum, QueryVS, and SumMe datasets are randomly divided into five splits, respectively.…”

Section: Evaluation and Analysismentioning

confidence: 99%

“…For each of them, 80% of the dataset is used for training, and the remaining for evaluation. F 1 -score [13,18,37,55] is adopted to measure the matching degree of the generated video summaries S i and the ground-truth video summaries Ŝi for video i. State-of-the-art comparisons.…”

Section: Evaluation and Analysismentioning

confidence: 99%

See 1 more Smart Citation

Causal Video Summarizer for Video Exploration

Huang

Yang

Chen³

et al. 2022

2022 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

The goal of video summarization is to automatically shorten videos such that it conveys the overall story without losing relevant information. In many application scenarios, improper video summarization can have a large impact. For example in forensics, the quality of the generated video summary will affect an investigator's judgment while in journalism it might yield undesired bias. Because of this, modeling explainability is a key concern. One of the best ways to address the explainability challenge is to uncover the causal relations that steer the process and lead to the result. Current machine learning-based video summarization algorithms learn optimal parameters but do not uncover causal relationships. Hence, they suffer from a relative lack of explainability. In this work, a Causal Explainer, dubbed Causalainer, is proposed to address this issue. Multiple meaningful random variables and their joint distributions are introduced to characterize the behaviors of key components in the problem of video summarization. In addition, helper distributions are introduced to enhance the effectiveness of model training. In visual-textual input scenarios, the extra input can decrease the model performance. A causal semantics extractor is designed to tackle this issue by effectively distilling the mutual information from the visual and textual inputs. Experimental results on commonly used benchmarks demonstrate that the proposed method achieves state-of-the-art performance while being more explainable.

show abstract

Spatial–temporal multi-scale interaction for few-shot video summarization

Li,

Zhan,

et al. 2025

Engineering Applications of Artificial Intelligence

View full text Add to dashboard Cite

Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer

Cited by 28 publications

References 78 publications

Transferable Video Moment Localization by Moment-Guided Query Prompting

Transferable Video Moment Localization by Moment-Guided Query Prompting

Causal Video Summarizer for Video Exploration

Spatial–temporal multi-scale interaction for few-shot video summarization

Contact Info

Product

Resources

About