Weak Supervision and Referring Attention for Temporal-Textual Association Learning

Fang, Zhiyuan; Kong, Shu; Wang, Zhe; Fowlkes, Charless; Yang, Yezhou

doi:10.48550/arxiv.2006.11747

Cited by 3 publications

(4 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare AutoTVG with other weakly-supervised counterparts by applying video moment generation method to Charades-STA and take the moment that has the maximum similarity with its caption as the final prediction. We observe that AutoTVG is able to achieve highly comparable results with CTF [7] and WSRA [11], and even surpasses SCN [29] by 4.2% at R@0.5 and CTF [7] by 1.1% at mIoU, which shows our video moment generation module provides reliable candidate moments. Thanks to the strong alignment between vision and language features from CLIP, we can obtain decent weakly-supervised results under a simple non-parametric matching strategy.…”

Section: Resultsmentioning

confidence: 64%

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

Zhang

Wang

et al. 2020

Natural Language Processing and Chinese Computing

View full text Add to dashboard Cite

The performance of Large Language Models (LLMs) in reasoning tasks depends heavily on prompt design, with Chain-of-Thought (CoT) and self-consistency being critical methods that enhance this ability. However, these methods do not fully exploit the answers generated by the LLM to guide subsequent responses. This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP), that enables automatic multiple interactions between users and LLMs by using previously generated answers as hints to progressively guide toward the correct answers. PHP is orthogonal to CoT and self-consistency, making it easy to combine with state-of-the-art techniques to further improve performance. We conducted an extensive and comprehensive evaluation to demonstrate the effectiveness of the proposed method. Our experimental results on six benchmarks show that combining CoT and self-consistency with PHP significantly improves accuracy while remaining highly efficient. For instance, with text-davinci-003, we observed a 4.2% improvement on GSM8K with greedy decoding compared to Complex CoT, and a 46.17% reduction in sample paths with self-consistency. With GPT-4 and PHP, we achieve state-of-the-art performances on SVAMP (91.9%), GSM8K (95.5%) and AQuA (79.9%).Preprint. Under review.

show abstract

Section: Resultsmentioning

confidence: 64%

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

Zhang

Wang

et al. 2020

Natural Language Processing and Chinese Computing

View full text Add to dashboard Cite

show abstract

“…For Weakly-Supervised Video Localization [7,25], due to the lack of accurate timestamp annotations, researchers try to promote models in learning the cross-modal correlation without the supervision of time stamps. To overcome this difficulty, many weakly supervised models use contrastive learning to use the similarity information between sentences and make full use of the combination information between video and text.…”

Section: Weakly Supervised Temporal Localizationmentioning

confidence: 99%

“…To demonstrate that learning more fine-grained phrase-level predictions is beneficial to improve model's generalization ability to new combinations of seen phrases (combinational generazation), we put forward a new dataset split on Charades-STA. Inspired by data splitting methods proposed in some weakly supervised settings [7,25], we aim to test the model's performance in this scenario: the data distributions of training set and testing set are different. We split the Charades-STA dataset as below to maximize the variance of phrases in the training section.…”

Section: Experiments Settingsmentioning

confidence: 99%

Phrase-level Prediction for Video Temporal Localization

Zheng

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Video temporal localization aims to locate a period that semantically matches a natural language query in a given untrimmed video. We empirically observe that although existing approaches gain steady progress on sentence localization, the performance of phrase localization is far from satisfactory. In principle, the phrase should be easier to localize as fewer combinations of visual concepts need to be considered; such incapability indicates that the existing models only capture the sentence annotation bias in the benchmark but lack sufficient understanding of the intrinsic relationship between simple visual and language concepts, thus the model generalization and interpretability is questioned. This paper proposes a unified framework that can deal with both sentence and phrase-level localization, namely Phrase Level Prediction Net (PLPNet). Specifically, based on the hypothesis that similar phrases tend to focus on similar video cues, while dissimilar ones should not, we build a contrastive mechanism to restrain phrase-level localization without fine-grained phrase boundary annotation required in training. Moreover, considering the sentence's flexibility and wide discrepancy among phrases, we propose a clustering-based batch sampler to ensure that contrastive learning can be conducted efficiently. Extensive experiments demonstrate that our method surpasses state-of-the-art methods of phrase-level temporal localization while maintaining high performance in sentence localization and boosting the model's interpretability and generalization capability. Our code is available at https://github.com/sizhelee/PLPNet. CCS CONCEPTS• Computing methodologies → Visual content-based indexing and retrieval; Activity recognition and understanding.

show abstract

“…Kervadec et al [25] use weak supervision in the form of object-word alignment as a pre-training task, Trott et al [53] use the counts of objects in an image as weak supervision to guide VQA for countingbased questions, Gokhale et al [16] use rules about logical connectives to augment training datasets for yes-no questions, and Zhao et al [65] use word-embeddings [36] to design an additional weak-supervision objective. Weak supervision from captions has also been recently used for visual grounding tasks [19,37,12,4].…”

Section: Related Workmentioning

confidence: 99%

Weakly Supervised Relative Spatial Reasoning for Visual Question Answering

Banerjee¹,

Gokhale²,

Yang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Vision-and-language (V&L) reasoning necessitates perception of visual concepts such as objects and actions, understanding semantics and language grounding, and reasoning about the interplay between the two modalities. One crucial aspect of visual reasoning is spatial understanding, which involves understanding relative locations of objects, i.e. implicitly learning the geometry of the scene. In this work, we evaluate the faithfulness of V&L models to such geometric understanding, by formulating the prediction of pair-wise relative locations of objects as a classification as well as a regression task. Our findings suggest that state-ofthe-art transformer-based V&L models lack sufficient abilities to excel at this task. Motivated by this, we design two objectives as proxies for 3D spatial reasoning (SR) -object centroid estimation, and relative position estimation, and train V&L with weak supervision from off-the-shelf depth estimators. This leads to considerable improvements in accuracy for the "GQA" visual question answering challenge (in fully supervised, few-shot, and O.O.D settings) as well as improvements in relative spatial reasoning. Code and data will be released here.

show abstract

Weak Supervision and Referring Attention for Temporal-Textual Association Learning

Cited by 3 publications

References 47 publications

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

Phrase-level Prediction for Video Temporal Localization

Weakly Supervised Relative Spatial Reasoning for Visual Question Answering

Contact Info

Product

Resources

About