A variational U‐Net for motion retargeting

Abstract

Motion retargeting is the process of copying motion from one character (source) to another (target) when the source and target body sizes and proportions (of arms, legs, torso, etc.) are different. The problem of automatic motion retargeting has been studied for several decades; however, the motion quality obtained with the application of current approaches is on occasion unrealistic. This is because previous methods, which are mainly based on numerical optimization, generally do not incorporate prior knowledge of the details and nuances of human movements. To address these issues, we present a novel human motion retargeting system using a deep learning framework with large-scale motion data to produce high-quality retargeted human motion. We establish a deep-learning-based motion retargeting system using a variational deep autoencoder combining the deep convolutional inverse graphics network (DC-IGN) and the U-Net. The DC-IGN is utilized for disentangling the motion of each body part, while the U-Net is employed to preserve details of the original motion. We conduct several experiments to validate the proposed motion retargeting system, and find that ours achieves better accuracy along with reduced computational burden when compared with the conventional motion retargeting approach and other neural network architectures.

UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons

“…However, it is also important to acknowledge that while these two problems can be technically decoupled, there may be complex interactions between them in practice. There are indeed a lot of works [31,35,54,77] on learning to retarget between different skeletons; or on learning cospeech gesture generation [6,20,50,52]. We are the first approach to attempt to integrate the both, and the integration of retargeting with speech-driven gestures can yield impressive results.…”

“…Ma et al [54] try to map multiple datasets to a defined skeleton, but it still partially relies on handcrafting and the results are still limited to a specific motion skeleton. Some work [31,35] try to retarget the motion of different skeletons by VAE, using standard convolution and pooling. However, unlike images or videos, different skeletons exhibit irregular connectivity.…”