Diverse Dance Synthesis via Keyframes with Transformer Controllers

Abstract

Existing keyframe-based motion synthesis mainly focuses on the generation of cyclic actions or short-term motion, such as walking, running, and transitions between close postures. However, these methods will significantly degrade the naturalness and diversity of the synthesized motion when dealing with complex and impromptu movements, e.g., dance performance and martial arts. In addition, current research lacks fine-grained control over the generated motion, which is essential for intelligent human-computer interaction and animation creation. In this paper, we propose a novel keyframe-based motion generation network based on multiple constraints, which can achieve diverse dance synthesis via learned knowledge. Specifically, the algorithm is mainly formulated based on the recurrent neural network (RNN) and the Transformer architecture. The backbone of our network is a hierarchical RNN module composed of two long short-term memory (LSTM) units, in which the first LSTM is utilized to embed the posture information of the historical frames into a latent space, and the second one is employed to predict the human posture for the next frame. Moreover, our framework contains two Transformer-based controllers, which are used to model the constraints of the root trajectory and the velocity factor respectively, so as to better utilize the temporal context of the frames and achieve fine-grained motion control. We verify the proposed approach on a dance dataset containing a wide range of contemporary dance. The results of three quantitative analyses validate the superiority of our algorithm. The video and qualitative experimental results demonstrate that the complex motion sequences generated by our algorithm can achieve diverse and smooth motion transitions between keyframes, even for long-term synthesis.

Foot‐constrained spatial‐temporal transformer for keyframe‐based complex motion synthesis

“…Dance. The dance dataset is collected by Aristidou et al 34 According to the division of Pan et al, 6 there are 123 pieces of contemporary dance and 93,347 motion frames in total. Among them, 98 segments with 93,347 frames are used as the training set and 25 segments with 20,897 frames are regarded as the test set.…”

“…Experiment on LaFAn1 dataset. We implement quantitative experiment on the LaFan1 dataset and make comparisons with the linear interpolation, as well as deep neural networks proposed by Harvey et al, 2 Pan et al 6 and Duan et al 3 The L2Q, L2P, and NPSS assessment metrics for the LAFAN1 dataset are evaluated. During the test, we only give the starting and ending frames of the keyframes and report the results of transition length of 5, 30, and 45 frames in Table 1.…”

“…Motion movements in this dataset are more complex and challenging. We contrast with the linear interpolation, as well as state‐of‐the‐art methods proposed by Harvey et al, 2 Pan et al 6 and Duan et al, 3 and report the L2P, L2Q, and NPSS results in Table 1. Motion performance with transition lengths of 10, 30, and 50 frames are assessed.…”

“…To evaluate the foot sliding artifacts, we perform qualitative and quantitative evaluations with state‐of‐the‐art methods. In qualitative comparison, we randomly choose a segment of motion generated by Harvey et al, 2 Pan et al, 6 Duan et al 3 and our method and depict the synthesized motions in Figure 4. The transition segment is 30 frames, we choose the middle 10 frames and visualize every 3 frames.…”

“…Lee et al 17 introduce a contact loss function to penalize foot sliding in training to deal with foot contact of non‐periodic motion. Pan et al 6 supervise the speed of footsteps in training to constrain the network to produce more realistic motion. Zou et al 18 propose a neural network‐based detector for localizing ground contact events on human feet by imposing physical constraints to optimize the dynamics of the entire motion.…”