Hierarchical Cross-Modal Graph Consistency Learning for Video-Text Retrieval

Jin, Weike; Zhao, Zhou; Zhang, Pengcheng; Zhu, Jieming; He, Xiuqiang; Zhuang, Yueting

doi:10.1145/3404835.3462974

Cited by 31 publications

(9 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, HGR (Chen et al 2020) is proposed to divide the sentence into three parts: events, action, and entity, with the hypothesis that all the captions own a relatively fixed hierarchical graph form. HCGC (Jin et al 2021) adopts the same sentence resolving strategy and introduces the hierar-chical cross-modal graph consistency learning into embedding space.…”

Section: Related Workmentioning

confidence: 99%

“…The heterogeneity of structures. This mainly lies in the impossibility of directly aligning the words in sentences with corresponding video frames (Jin et al 2021). Singlestream or two-stream structures are applied to treat text and video as two independent parts for early or late fusion, which ignore the internal relevancy between frames and words, resulting in that the models require massive data to reach decent performance.…”

Section: Introduction Motivationmentioning

confidence: 99%

“…Previous work has taken a similar approach, either by simple part-of-speech tagging or by wielding multi-dimensional features on the video. HGR (Chen et al 2020) and HCGC (Jin et al 2021) hypothesize that a text can be constructed into a hierarchical semantic graph structure, where lie sentence, action, entity embedding in the top, second, third level node, respectively. T2VLAD (Wang, Zhu, and Yang 2021) extracts features from the aspects of scene and action, and performs similarity matching with the representations of each local token and the global sentence, while HiT (Liu et al 2021) conducts cross-matching between featurelevel and semantic-level embedding.…”

Section: Introduction Motivationmentioning

confidence: 99%

See 2 more Smart Citations

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Cheng¹,

Lin²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Employing large-scale pre-trained model CLIP to conduct video-text retrieval task (VTR) has become a new trend, which exceeds previous VTR methods. Though, due to the heterogeneity of structures and contents between video and text, previous CLIP-based models are prone to overfitting in the training phase, resulting in relatively poor retrieval performance. In this paper, we propose a multi-stream Corpus Alignment network with single gate Mixture-of-Experts (CAMoE) and a novel Dual Softmax Loss (DSL) to solve the two heterogeneity. The CAMoE employs Mixture-of-Experts (MoE) to extract multi-perspective video representations, including action, entity, scene, etc., then align them with the corresponding part of the text. In this stage, we conduct massive explorations towards the feature extraction module and feature alignment module, and conclude an efficient VTR framework. DSL is proposed to avoid the oneway optimum-match which occurs in previous contrastive methods. Introducing the intrinsic prior of each pair in a batch, DSL serves as a reviser to correct the similarity matrix and achieves the dual optimal match. DSL is easy to implement with only one-line code but improves significantly. The results show that the proposed CAMoE and DSL are of strong efficiency, and each of them is capable of achieving State-of-The-Art (SOTA) individually on various benchmarks such as MSR-VTT, MSVD, and LSMDC. Further, with both of them, the performance is advanced to a great extent, surpassing the previous SOTA methods for around 4.6% R@1 in MSR-VTT. The code will be available soon at https://github.com/starmemda/CAMoE/

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introduction Motivationmentioning

confidence: 99%

Section: Introduction Motivationmentioning

confidence: 99%

See 1 more Smart Citation

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Cheng¹,

Lin²,

Wu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Inspired by the investigation of graph consistency in [16,55], to explore the agreement property of each item based on the augmented views, we define knowledge graph structure consistency 𝑐 𝑖 of item 𝑖 with the agreement between the representations encoded from different views as follows:…”

Section: Agreement Between Augmented Structural Viewsmentioning

confidence: 99%

Knowledge Graph Contrastive Learning for Recommendation

Yang,

Huang,

Xia

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge Graphs (KGs) have been utilized as useful side information to improve recommendation quality. In those recommender systems, knowledge graph information often contains fruitful facts and inherent semantic relatedness among items. However, the success of such methods relies on the high quality knowledge graphs, and may not learn quality representations with two challenges: i) The long-tail distribution of entities results in sparse supervision signals for KG-enhanced item representation; ii) Real-world knowledge graphs are often noisy and contain topic-irrelevant connections between items and entities. Such KG sparsity and noise make the item-entity dependent relations deviate from reflecting their true characteristics, which significantly amplifies the noise effect and hinders the accurate representation of user's preference.To fill this research gap, we design a general Knowledge Graph Contrastive Learning framework (KGCL) that alleviates the information noise for knowledge graph-enhanced recommender systems. Specifically, we propose a knowledge graph augmentation schema to suppress KG noise in information aggregation, and derive more robust knowledge-aware representations for items. In addition, we exploit additional supervision signals from the KG augmentation process to guide a cross-view contrastive learning paradigm, giving a greater role to unbiased user-item interactions in gradient descent and further suppressing the noise. Extensive experiments on three public datasets demonstrate the consistent superiority of our KGCL over state-of-the-art techniques. KGCL also achieves strong performance in recommendation scenarios with sparse user-item interactions, long-tail and noisy KG entities. Our implementation codes are available at https://github.com/yuh-yang/KGCL-SIGIR22. CCS CONCEPTS• Information systems → Recommender systems.

show abstract

“…DNNs are known to have enticing representation capability and have the natural strength to capture comprehensive relations [76] over different entities (e.g., items, users, interactions). Recently, there are works that explore advanced techniques, e.g., memory networks [53], attention mechanisms [56,79], and graph neural networks [9,26,31,36,81] for sequential recommendation [6,23,29,54,61,67,72]. Typically, MIND [32] adopts the dynamic routing mechanism to aggregate users' behaviors into multiple interest vectors.…”

Section: Related Work 21 Sequential Recommendationmentioning

confidence: 99%

CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation

Zhang

Yao

Zhao

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

107

View full text Add to dashboard Cite

Learning user representations based on historical behaviors lies at the core of modern recommender systems. Recent advances in sequential recommenders have convincingly demonstrated high capability in extracting effective user representations from the given behavior sequences. Despite significant progress, we argue that solely modeling the observational behaviors sequences may end up with a brittle and unstable system due to the noisy and sparse nature of user interactions logged. In this paper, we propose to learn accurate and robust user representations, which are required to be less sensitive to (attack on) noisy behaviors and trust more on the indispensable ones, by modeling counterfactual data distribution. Specifically, given an observed behavior sequence, the proposed CauseRec framework identifies dispensable and indispensable concepts at both the fine-grained item level and the abstract interest level. CauseRec conditionally samples user concept sequences from the counterfactual data distributions by replacing dispensable and indispensable concepts within the original concept sequence. With user representations obtained from the synthesized user sequences, CauseRec performs contrastive user representation learning by contrasting the counterfactual with the observational. We conduct extensive experiments on real-world public recommendation benchmarks and justify the effectiveness of CauseRec with multiaspects model analysis. The results demonstrate that the proposed CauseRec outperforms state-of-the-art sequential recommenders by learning accurate and robust user representations . CCS CONCEPTS• Information systems → Recommender systems.

show abstract

Hierarchical Cross-Modal Graph Consistency Learning for Video-Text Retrieval

Cited by 31 publications

References 37 publications

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

Knowledge Graph Contrastive Learning for Recommendation

CauseRec: Counterfactual User Sequence Synthesis for Sequential Recommendation

Contact Info

Product

Resources

About