Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Lei, Chenyi; Luo, Shixian; Liu, Yong; He, Wanggui; Wang, Jiamang; Wang, Guoxin; Tang, Haitao; Chen, Miao; Li, Houqiang

doi:10.1145/3474085.3475431

Cited by 30 publications

(14 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The visual entailment result of ALBEF on SNLI-VE dataset is shown in TR@5 TR@10 IR@1 IR@5 IR@10 (b) The impact of 𝛼 2 It is shown that as 𝛼 1 > 0 and 𝛼 2 > 0, the attack performance becomes stronger. This demonstrates the importance of the second term in Equation ( 6) and Equation (7).…”

Section: Ablation Studymentioning

confidence: 81%

“…It is shown that as 𝛼 1 > 0 and 𝛼 2 > 0, the attack performance becomes stronger. This demonstrates the importance of the second term in Equation (6) and Equation(7). The results are comparable when 𝛼 1 ≥ 1 and 𝛼 2 ≥ 1, indicating that Co-Attack is not sensitive to hyper-parameters and does not require elaborate tuning for the hyper-parameters.…”

mentioning

confidence: 80%

“…We conducted the ablation experiments to study the impact of 𝛼 1 in Equation ( 6) and 𝛼 2 in Equation (7). We adjusted 𝛼 1 and 𝛼 2 within the range of [0, 5] with a step of 1 and examined their influences on ALBEF and CLIP ViT , respectively.…”

Section: Ablation Studymentioning

confidence: 99%

See 2 more Smart Citations

Towards Adversarial Attack on Vision-Language Pre-training Models

Zhang¹,

Yi²,

Sang³

2022

Preprint

View full text Add to dashboard Cite

While vision-language pre-training model (VLP) has shown revolutionary improvements on various vision-language (V+L) tasks, the studies regarding its adversarial robustness remain largely unexplored. This paper studied the adversarial attack on popular VLP models and V+L tasks. First, we analyzed the performance of adversarial attacks under different settings. By examining the influence of different perturbed objects and attack targets, we concluded some key observations as guidance on both designing strong multimodal adversarial attack and constructing robust VLP models. Second, we proposed a novel multimodal attack method on the VLP models called Collaborative Multimodal Adversarial Attack (Co-Attack), which collectively carries out the attacks on the image modality and the text modality. Experimental results demonstrated that the proposed method achieves improved attack performances on different V+L downstream tasks and VLP models. The analysis observations and novel attack method hopefully provide new understanding into the adversarial robustness of VLP models, so as to contribute their safe and reliable deployment in more real-world scenarios. CCS CONCEPTS• Computing methodologies → Artificial intelligence.

show abstract

Section: Ablation Studymentioning

confidence: 81%

mentioning

confidence: 80%

See 1 more Smart Citation

Towards Adversarial Attack on Vision-Language Pre-training Models

Zhang¹,

Yi²,

Sang³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The studies of ClipBERT (Lei et al, 2021b) and Frozen (Bain et al, 2021) demonstrate that imagetext pre-training is effective in improving downstream video-text performance. Recent efforts in image-text modeling have also shown that, when scaled up to hundreds of millions (Radford et al, 2021) or even billions (Li et al, 2021a) of image-text pairs, image-text models can achieve stateof-the-art results on various video-text tasks, including text-to-video retrieval (Luo et al, 2021a;Yuan et al, 2021;Yu et al, 2022a), video question answering (Alayrac et al, 2022), and video captioning (Tang et al, 2021a;Wang et al, 2022d).…”

Section: Transferring Image-text Models To Video-text Tasksmentioning

confidence: 99%

“…Due to the missing of large-scale non-English video-text datasets for both pre-training and downstream evaluation, video-text models in non-English languages are less explored. As initial attempts, Lei et al (2021a)…”

Section: Multi-lingual Vlp For Video-text Tasksmentioning

confidence: 99%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Gan¹,

Fu²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper surveys vision-language pre-training (VLP) methods for multimodal intelligence that have been developed in the last few years. We group these approaches into three categories: (i) VLP for image-text tasks, such as image captioning, image-text retrieval, visual question answering, and visual grounding; (ii) VLP for core computer vision tasks, such as (open-set) image classification, object detection, and segmentation; and (iii) VLP for video-text tasks, such as video captioning, video-text retrieval, and video question answering. For each category, we present a comprehensive review of state-of-the-art methods, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies. In addition, for each category, we discuss advanced topics being actively explored in the research community, such as big foundation models, unified modeling, in-context few-shot learning, knowledge, robustness, and computer vision in the wild, to name a few.♠ Zhe Gan and Jianfeng Gao initiated the project. Zhe Gan and Linjie Li took lead in the writing of Chapter 1. Linjie Li and Jianfeng Gao took lead in the writing of Chapter 2. Zhe Gan further took lead in the writing of Chapter 3 and 7. Chunyuan Li took lead in the writing of Chapter 4. Linjie Li further took lead in the writing of Chapter 5. Lijuan Wang and Zicheng Liu took lead in the writing of Chapter 6. All the authors provided project advice, and contributed to paper editing and proofreading.

show abstract

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

Zhang

Wang

et al. 2020

Natural Language Processing and Chinese Computing

View full text Add to dashboard Cite

The performance of Large Language Models (LLMs) in reasoning tasks depends heavily on prompt design, with Chain-of-Thought (CoT) and self-consistency being critical methods that enhance this ability. However, these methods do not fully exploit the answers generated by the LLM to guide subsequent responses. This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP), that enables automatic multiple interactions between users and LLMs by using previously generated answers as hints to progressively guide toward the correct answers. PHP is orthogonal to CoT and self-consistency, making it easy to combine with state-of-the-art techniques to further improve performance. We conducted an extensive and comprehensive evaluation to demonstrate the effectiveness of the proposed method. Our experimental results on six benchmarks show that combining CoT and self-consistency with PHP significantly improves accuracy while remaining highly efficient. For instance, with text-davinci-003, we observed a 4.2% improvement on GSM8K with greedy decoding compared to Complex CoT, and a 46.17% reduction in sample paths with self-consistency. With GPT-4 and PHP, we achieve state-of-the-art performances on SVAMP (91.9%), GSM8K (95.5%) and AQuA (79.9%).Preprint. Under review.

show abstract

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Cited by 30 publications

References 18 publications

Towards Adversarial Attack on Vision-Language Pre-training Models

Towards Adversarial Attack on Vision-Language Pre-training Models

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

The Solution of Huawei Cloud & Noah’s Ark Lab to the NLPCC-2020 Challenge: Light Pre-Training Chinese Language Model for NLP Task

Contact Info

Product

Resources

About