Breathing Life Into Sketches Using Text-to-Video Priors

Gal, Rinon; Vinker, Yael; Alaluf, Yuval; Bermano, Amit; Cohen-Or, Daniel; Shamir, Ariel; Chechik, Gal

doi:10.1109/cvpr52733.2024.00414

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Other1

Article1

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

Li,

Shi,

et al. 2024

SIGGRAPH Asia 2024 Conference Papers

View full text Add to dashboard Cite

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

Li,

Shi,

et al. 2024

SIGGRAPH Asia 2024 Conference Papers

View full text Add to dashboard Cite

ToonCrafter: Generative Cartoon Interpolation

Xing,

Liu,

Xia

et al. 2024

ACM Trans. Graph.

View full text Add to dashboard Cite

We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors. Code and model weights are available at https://doubiiu.github.io/projects/ToonCrafter

show abstract

Breathing Life Into Sketches Using Text-to-Video Priors

Cited by 2 publications

References 56 publications

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generation

ToonCrafter: Generative Cartoon Interpolation

Contact Info

Product

Resources

About