Make It Move: Controllable Image-to-Video Generation with Text Descriptions

Hu, Yaosi; Luo, Cheng; Chen, Zhenzhong

doi:10.48550/arxiv.2112.02815

Cited by 1 publication

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, TVP could be regarded as a more controllable and plausible task. The concurrent work [24] proposes a similar setting and adopts a VQ-VAE based framework, which is totally different from our GAN-based inference network. Besides, this work [24] uses the first image and text to produce a motion anchor, thus guiding the generation for all subsequent frames.…”

Section: B Image-to-video Generationmentioning

confidence: 99%

“…The concurrent work [24] proposes a similar setting and adopts a VQ-VAE based framework, which is totally different from our GAN-based inference network. Besides, this work [24] uses the first image and text to produce a motion anchor, thus guiding the generation for all subsequent frames. In contrast, our framework fully explores the inference ability of text on motion information to generate step-wise embeddings specifically for each subsequent frame.…”

Section: B Image-to-video Generationmentioning

confidence: 99%

See 1 more Smart Citation

Text-driven Video Prediction

Xue¹,

Chen²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Current video generation models usually convert signals indicating appearance and motion received from inputs (e.g., image, text) or latent spaces (e.g., noise vectors) into consecutive frames, fulfilling a stochastic generation process for the uncertainty introduced by latent code sampling. However, this generation pattern lacks deterministic constraints for both appearance and motion, leading to uncontrollable and undesirable outcomes. To this end, we propose a new task called Text-driven Video Prediction (TVP). Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. Specifically, appearance and motion components are provided by the image and caption separately. The key to addressing the TVP task depends on fully exploring the underlying motion information in text descriptions, thus facilitating plausible video generation. In fact, this task is intrinsically a cause-and-effect problem, as the text content directly influences the motion changes of frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM), producing step-wise embeddings to regulate motion inference for subsequent frames. In particular, a refinement mechanism incorporating global motion semantics guarantees coherent generation. Extensive experiments are conducted on Something-Something V2 and Single Moving MNIST datasets. Experimental results demonstrate that our model achieves better results over other baselines, verifying the effectiveness of the proposed framework.

show abstract

Section: B Image-to-video Generationmentioning

confidence: 99%

Section: B Image-to-video Generationmentioning

confidence: 99%