Towards Accurate Generative Models of Video: A New Metric &amp; Challenges

Unterthiner, Thomas; Steenkiste, Sjoerd van; Kurach, Karol; Marinier, Raphaël; Michalski, Marcin; Gelly, Sylvain

doi:10.48550/arxiv.1812.01717

Cited by 71 publications

(109 citation statements)

References 0 publications

Supporting

Mentioning

109

Contrasting

Order By: Relevance

“…To quantitatively evaluate MAGE, we apply conventional pixel-based similarity metrics SSIM and PSNR for deterministic video generation. We also report several perceptual similarity metrics including imagelevel Fréchet Inception Distance (FID) [13] and Learned Perceptual Image Patch Similarity (LPIPS) [9], as well as video-level Fréchet-Video-Distance (FVD) [34]. To evaluate the diversity of generated videos given ambiguous text, following previous work [8], we measure the average mutual distance of generated video sequences in the feature space of both VGG-16 [31] and I3D [32] network.…”

Section: B Quantitative Resultsmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Imageto-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Source code for model and datasets will be available soon.* This work was done while Yaosi Hu was an intern at MSRA.

show abstract

Section: B Quantitative Resultsmentioning

confidence: 99%

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu¹,

Luo²,

Chen³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Although given only one frame as condition (Cond. ), N ÜWA still significantly pushes the state-of-the-art FVD [38] score from 94±2 to 86.9.…”

Section: Comparison With State-of-the-artmentioning

confidence: 92%

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Wu¹,

Ji²,

Ji³

et al. 2021

Preprint

View full text Add to dashboard Cite

“…However, perfect image quality and static consistency can be achieved by simply not animating anything at all and they cannot reflect the quality of generated videos. Thus we use Fréchet Video Distance [25] to measure the quality of generated videos, especially on their motion. User studies are further conducted to evaluate their quality from human sense.…”

Section: Methodsmentioning

confidence: 99%

Learning Fine-Grained Motion Embedding for Landscape Animation

Xue

Liu

Yang

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper we focus on landscape animation, which aims to generate time-lapse videos from a single landscape image. Motion is crucial for landscape animation as it determines how objects move in videos. Existing methods are able to generate appealing videos by learning motion from real time-lapse videos. However, current methods suffer from inaccurate motion generation, which leads to unrealistic video results. To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation. Our model consists of two parts: (1) a motion encoder which embeds time-lapse motion in a fine-grained way. (2) a motion generator which generates realistic motion to animate input images. To train and evaluate on diverse time-lapse videos, we build the largest high-resolution Time-lapse video dataset with Diverse scenes, namely Time-lapse-D, which includes 16,874 video clips with over 10 million frames. Quantitative and qualitative experimental results demonstrate the superiority of our method. In particular, our method achieves relative improvements by 19% on LIPIS and 5.6% on FVD compared with state-of-the-art methods on our dataset. A user study carried out with 700 human subjects shows that our approach visually outperforms existing methods by a large margin.

show abstract

Towards Accurate Generative Models of Video: A New Metric & Challenges

Cited by 71 publications

References 0 publications

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Make It Move: Controllable Image-to-Video Generation with Text Descriptions

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Learning Fine-Grained Motion Embedding for Landscape Animation

Contact Info

Product

Resources

About