Structure-Aware Motion Transfer with Deformable Anchor Model

Tao, Jiale; Wang, Biao; Xu, Borun; Ge, Tiezheng; Jiang, Yuning; Wen, Li; Duan, Lixin

doi:10.1109/cvpr52688.2022.00362

Cited by 28 publications

(9 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A key focus area of such works is to design appropriate motion representations for animation [35,52,55,66]. A number of improved representations have been proposed, such as those setting additional constraints on a kinematic tree [59], and thin-plate spline motion modelling [81]. A further work, titled Latent Image Animator [64], learned a latent space for possible motions.…”

Section: Related Workmentioning

confidence: 99%

“…Initially, such transformations were modeled using a simple set of sparse keypoints. Further works improved the motion representation [52,55], learned latent motion dictionaries [64], kinematic chains [59] or used thin-plate spline transformations [81]. However, broadly speaking, all such works propose 2D motion representations, warping the pixels or features of the input image such that they correspond to the pose of a given driving image.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Unsupervised Volumetric Animation

Siarohin¹,

Menapace²,

Skorokhodov³

et al. 2023

Preprint

View full text Add to dashboard Cite

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Unsupervised Volumetric Animation

Siarohin¹,

Menapace²,

Skorokhodov³

et al. 2023

Preprint

View full text Add to dashboard Cite

“…Unsupervised methods [34,35,37,40] have been recently proposed to address the above limitation. These approaches typically leverage a large amount of easy-to-obtain unlabeled web videos and design image reconstruction losses to learn intermediate motion representations (e.g., keypoints and affine matrices).…”

Section: Related Workmentioning

confidence: 99%

Motion Transformer for Unsupervised Image Animation

Tao¹,

Wang²,

Ge³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Image animation aims to animate a source image by using motion learned from a driving video. Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information, such as motion keypoints and corresponding local transformations. However, these CNN based methods do not explicitly model the interactions between motions; as a result, the important underlying motion relationship may be neglected, which can potentially lead to noticeable artifacts being produced in the generated animation video. To this end, we propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer. More specifically, we introduce two types of tokens in our proposed method: i) image tokens formed from patch features and corresponding position encoding; and ii) motion tokens encoded with motion information. Both types of tokens are sent into vision transformers to promote underlying interactions between them through multi-head self attention blocks. By adopting this process, the motion information can be better learned to boost the model performance. The final embedded motion tokens are then used to predict the corresponding motion keypoints and local transformations. Extensive experiments on benchmark datasets show that our proposed method achieves promising results to the state-of-the-art baselines. Our source code will be public available.

show abstract

“…The model-free approaches [30,31,19,32,34] does not rely on pre-trained third-party models, and extend the model-based method to arbitrary objects. Aliaksandr et al [30] proposed a model-free motion transfer model Monky-Net that can apply motion transfer on arbitrary objects with an unsupervised key point detector trained by reconstruction loss [18].…”

Section: Related Workmentioning

confidence: 99%

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

Xu¹,

Wang²,

Deng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Motion transfer aims to transfer the motion of a driving video to a source image. When there are considerable differences between object in the driving video and that in the source image, traditional single domain motion transfer approaches often produce notable artifacts; for example, the synthesized image may fail to preserve the human shape of the source image (cf . Fig. 1 (a)). To address this issue, in this work, we propose a Motion and Appearance Adaptation (MAA) approach for cross-domain motion transfer, in which we regularize the object in the synthesized image to capture the motion of the object in the driving frame, while still preserving the shape and appearance of the object in the source image. On one hand, considering the object shapes of the synthesized image and the driving frame might be different, we design a shape-invariant motion adaptation module that enforces the consistency of the angles of object parts in two images to capture the motion information. On the other hand, we introduce a structure-guided appearance consistency module designed to regularize the similarity between the corresponding patches of the synthesized image and the source image without affecting the learned motion in the synthesized image. Our proposed MAA model can be trained in an end-to-end manner with a cyclic reconstruction loss, and ultimately produces a satisfactory motion transfer result (cf . Fig. 1 (b)). We conduct extensive experiments on human dancing dataset Mixamo-Video to Fashion-Video and human face dataset Vox-Celeb to Cufs; on both of these, our MAA model outperforms existing methods both quantitatively and qualitatively.

show abstract

Structure-Aware Motion Transfer with Deformable Anchor Model

Cited by 28 publications

References 33 publications

Unsupervised Volumetric Animation

Unsupervised Volumetric Animation

Motion Transformer for Unsupervised Image Animation

Motion and Appearance Adaptation for Cross-Domain Motion Transfer

Contact Info

Product

Resources

About