TGIF: A New Dataset and Benchmark on Animated GIF Description

Li, Yuncheng; Song, Yang; Cao, Liangliang; Tetreault, Joel; Goldberg, Larry; Jaimes, Alejandro; Luo, Jiebo

doi:10.48550/arxiv.1604.02748

Cited by 11 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For imagelanguage pre-training, there exist numerous large-scale and highquality datasets with over 100 thousand unique images, including English datasets [15,24] and Chinese datasets [10,23] (10 millions level). However, there are only three datasets including over 100 thousand unique videos for video-language pre-training, i.e., TGIF [22], AutoGIF [29], and HowTo100M [28]. All TGIF, AutoGIF, and HowTo100M are English datasets and crawled from open websites.…”

Section: Related Workmentioning

confidence: 99%

“…To empower the pre-trained model with Chinese knowledge and evaluate the effectiveness of the pre-training mechanism on Chinese data, we build a video-language pre-training dataset, namely Alivol-10M, from one of the world's largest e-commerce platforms. Existing video-language datasets are usually collected from open websites [6,22,22,28,43], where the data are freely created by Table 2: Results of cross-modal video retrieval tasks. R@N denotes the recall of top-N predictions.…”

Section: Pre-training Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Lei¹,

Luo²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The pre-trained neural models have recently achieved impressive performances in understanding multimodal content. However, it is still very challenging to pre-train neural models for video and language understanding, especially for Chinese video-language data, due to the following reasons. Firstly, existing video-language pretraining algorithms mainly focus on the co-occurrence of words and video frames, but ignore other valuable semantic and structure information of video-language content, e.g., sequential order and spatiotemporal relationships. Secondly, there exist conflicts between video sentence alignment and other proxy tasks. Thirdly, there is a lack of large-scale and high-quality Chinese video-language datasets (e.g., including 10 million unique videos), which are the fundamental success conditions for pre-training techniques.In this work, we propose a novel video-language understanding framework named Victor, which stands for VIdeo-language understanding via Contrastive mulTimOdal pRe-training. Besides general proxy tasks such as masked language modeling, Victor constructs several novel proxy tasks under the contrastive learning paradigm, making the model be more robust and able to capture more complex multimodal semantic and structural relationships from different perspectives. Victor is trained on a large-scale Chinese video-language dataset, including over 10 million complete videos with corresponding high-quality textual descriptions. We apply the pre-trained Victor model to a series of downstream applications and demonstrate its superior performances, comparing against the state-of-the-art pre-training methods such as VideoBERT and UniVL. The codes and trained checkpoints will be publicly available to nourish further developments of the research community.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Pre-training Datasetmentioning

confidence: 99%

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Lei¹,

Luo²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

DeepStory: Video Story QA by Deep Embedded Memory Networks

Kim

Heo

Choi

et al. 2017

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence

123

View full text Add to dashboard Cite

Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based attention model uses the long-term memory to recall the best question-story-answer triplet by focusing on specific words containing key information. We trained the DEMN on a novel QA dataset of children's cartoon video series, Pororo. The dataset contains 16,066 scene-dialogue pairs of 20.5-hour videos, 27,328 fine-grained sentences for scene description, and 8,913 story-related QA pairs. Our experimental results show that the DEMN outperforms other QA models. This is mainly due to 1) the reconstruction of video stories in a scene-dialogue combined form that utilize the latent embedding and 2) attention. DEMN also achieved state-of-the-art results on the MovieQA benchmark.

show abstract

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Jang

Song

et al. 2017

Preprint

View full text Add to dashboard Cite

TGIF: A New Dataset and Benchmark on Animated GIF Description

Cited by 11 publications

References 36 publications

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training

DeepStory: Video Story QA by Deep Embedded Memory Networks

TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering

Contact Info

Product

Resources

About